Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upGitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
GnuTLS breaks HTTPS connections with YouTube #13532
Comments
|
I confirm, I have a similar error. Appeared 2-3 days ago |
|
i am not sure which side is breaking youtube-dl,ffmpeg or its an problem with hls |
|
Oops https://www.youtube.com/watch?v=Cx1uJUVCLto has been removed. Any other example? |
|
i see the problem is with all live stream and videos, 2nd example is an live stream. youtube-dl https://www.youtube.com/watch?v=f3dvRuzjLHE -f 93 -v>1.tt |
|
@yan12125 i even have upgraded youtube-dl i.e 2nd july`17 so the problem still exists in the new version too.As I noticed both times ffmpeg throwed error as "ffmpeg exited with code 1". So do you have an idea what the flag 1 means or does it mean ffmpeg returning 0 is ffmpeg sucessfully transcoded/converted the media format and 1 means its exited with failure i.e exit(1) |
|
YouTube says https://www.youtube.com/watch?v=f3dvRuzjLHE is unavailable. Seems this bug occurs on videos that are to be gone? |
|
@yan12125 i surely agree because they are live feeds,once the live feed is over youtube will soon began to merge all the peices/fragment of the file.So it means youtube-dl can`t download live videos. |
|
Oh thanks I can reproduce it. There are indeed endless "The TLS connection was non-properly terminated." errors from ffmpeg. It doesn't seem to affect the downloaded file as I can watch the By the way, avconv doesn't report such errors: |
|
Update: the culprit is GnuTLS rather than ffmpeg. If I rebulid ffmpeg with OpenSSL, everything works: |
yan12125
changed the title
|
yes got it,but still did not understand lot of things though as "ffmpeg -protocols" dumped these protocols installed "Supported file protocols: |
|
@yan12125 I have not tested it against live feeds as I have exhausted by daily internet quota.So would try after 4 hours when the next day quota is started |
|
@yan12125 i simply added gnutls-bin i.e "sudo apt install gnutls-bin" and found it to be working fine youtube-dl https://www.youtube.com/watch?v=PmCkBPm7ICk |
|
@yan12125 i think to close the issue as its geeting external but I am unsure wheter to modify the ffmpeg.py file to have --enable-gnutls as an parameter i.e https://github.com/rg3/youtube-dl/blob/master/youtube_dl/postprocessor/ffmpeg.py |
|
quite frankly i have no idea about parameter in ffmpeg |
|
FYI: --enable-openssl is used when compiling ffmpeg. Change the step 3 of method 1 in #13532 (comment) from: to |
|
needed a way to fix ffmpeg.py residing in youtube-dl,less worried with ffmpeg as both --enable-gnutls and --enable-openssl failed due to system library issue which i just fixed with an additional package |
|
Do you mean those errors? I don't think modifying ffmpeg.py can fix it. You'll need to build a custom ffmpeg. |
|
@yan12125 i kept on digging in ffmpeg.py to try to understand how the internal call works,well turns out it can be expliclity be forced.PostProcessor can handle it.Just after line 66 an custom error flag can be given inside class FFmpegVideoConvertorPP(FFmpegPostProcessor), and let the process handle it via cmd at line 192.Just not able to figure out which line generates verbose line 162 or 197.So that the flag value can be put here. |
|
@siddht1 Could you paste relevant code lines? I'm not sure whether I have the same copy of ffmpeg.py as you. I always use git-master, which is newer than the latest stable release. |
|
@yan12125 my copy is the same as the current release version 2017.07.02 ,still i am copy pasting the code and providing a general idea of my approach if self._downloader:
cmd = ([encodeFilename(self.executable, True), encodeArgument('-y')] + #this is line 192 clearly i can see this would be using subprocess or process so based on flag value it will either set it to 1 i.e returned with error,2 for handling the gnutls. example 1: example 2: action_flag() |
|
@yan12125 i think you got an idea what i m trying to do |
|
still https://raw.githubusercontent.com/rg3/youtube-dl/master/youtube_dl/postprocessor/ffmpeg.py is eqivalent my ffmpeg.py |
|
Seems you want to specify --gnutls when youtube-dl runs ffmpeg? That won't work as openssl/gnutls is determined during compile time, not run time. By the way, if you want to feed additional flags to ffmpeg, just use --external-downloader-args. For example: |
|
@yan12125 how does |
|
|
|
@yan12125 thanks,as this issue is now clearly not handled by ffmpeg.py,closing the issue,can you just point out where does these verbose come from, |
|
Did you mean "The TLS connection was non-properly terminated" messages? Those are from GnuTLS. Also, it affects downloading performance besides extra verbosity. |
|
@yan12125 yes those message,ffmpeg.py must have that |
|
Use the fallback operator /: In the future, please open a new issue or ask on IRC for non-issues. Please don't leave unrelated contents on issues. |
Please follow the guide below
xinto all the boxes [ ] relevant to your issue (like that [x])Make sure you are using the latest version: run
youtube-dl --versionand ensure your version is 2017.06.25. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.Before submitting an issue make sure you have:
What is the purpose of your issue?
The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue
If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:
Add
-vflag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):
Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
Description of your issue, suggested solution and other information
Explanation of your issue in arbitrary form goes here. Please make sure the description is worded well enough to be understood. Provide as much context and examples as possible.
If work on your issue requires account credentials please provide them or explain how one can obtain them.
Hey there I was just downloading a youtube video but youtube-dl kept giving error code,this is the whole verbose of the problem i faced: youtube-dl https://www.youtube.com/watch?v=Cx1uJUVCLto -f 93 -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://www.youtube.com/watch?v=Cx1uJUVCLto', u'-f', u'93', u'-v']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.06.25
[debug] Python version 2.7.12 - Linux-4.10.0-21-generic-x86_64-with-LinuxMint-18.1-serena
[debug] exe versions: avconv 2.8.11-0ubuntu0.16.04.1, avprobe 2.8.11-0ubuntu0.16.04.1, ffmpeg 2.8.11-0ubuntu0.16.04.1, ffprobe 2.8.11-0ubuntu0.16.04.1
[debug] Proxy map: {}
[youtube] Cx1uJUVCLto: Downloading webpage
[youtube] Cx1uJUVCLto: Downloading video info webpage
[youtube] Cx1uJUVCLto: Extracting video information
[youtube] Cx1uJUVCLto: Downloading m3u8 information
[youtube] Cx1uJUVCLto: Downloading MPD manifest
[debug] Invoking downloader on u'https://manifest.googlevideo.com/api/manifest/hls_playlist/id/Cx1uJUVCLto.1/itag/93/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D134/hls_chunk_host/r1---sn-gwpa-gq2l.googlevideo.com/ei/KuxWWdW8NsaNoAP8rK1A/gcr/in/playlist_type/DVR/mm/32/mn/sn-gwpa-gq2l/ms/lv/mv/m/pl/22/dover/6/mt/1498868718/ip/47.15.9.230/ipbits/0/expire/1498890377/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,ei,gcr,playlist_type,mm,mn,ms,mv,pl/signature/96E2DB8417DEF3A420ECC42CF17CCD688B310FDD.696630EB61AFCFCE3968BA1A1FC8397AC11A8668/key/dg_yt0/playlist/index.m3u8'
[download] Destination: Pokemon Sun and Moon Episode 32 English Sub-Cx1uJUVCLto.mp4
[debug] avconv command line: avconv -y -loglevel verbose -headers 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,;q=0.7
Accept-Language: en-us,en;q=0.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/*;q=0.8
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)
' -i 'https://manifest.googlevideo.com/api/manifest/hls_playlist/id/Cx1uJUVCLto.1/itag/93/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D134/hls_chunk_host/r1---sn-gwpa-gq2l.googlevideo.com/ei/KuxWWdW8NsaNoAP8rK1A/gcr/in/playlist_type/DVR/mm/32/mn/sn-gwpa-gq2l/ms/lv/mv/m/pl/22/dover/6/mt/1498868718/ip/47.15.9.230/ipbits/0/expire/1498890377/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,ei,gcr,playlist_type,mm,mn,ms,mv,pl/signature/96E2DB8417DEF3A420ECC42CF17CCD688B310FDD.696630EB61AFCFCE3968BA1A1FC8397AC11A8668/key/dg_yt0/playlist/index.m3u8' -c copy -f mp4 'file:Pokemon Sun and Moon Episode 32 English Sub-Cx1uJUVCLto.mp4.part'
ffmpeg version 2.8.11-0ubuntu0.16.04.1 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 20160609
configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
WARNING: library configuration mismatch
avcodec configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv --enable-version3 --disable-doc --disable-programs --disable-avdevice --disable-avfilter --disable-avformat --disable-avresample --disable-postproc --disable-swscale --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libvo_aacenc --enable-libvo_amrwbenc
libavutil 54. 31.100 / 54. 31.100
libavcodec 56. 60.100 / 56. 60.100
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 40.101 / 5. 40.101
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100
[tls @ 0x1460ea0] The TLS connection was non-properly terminated.
[tls @ 0x1460ea0] The specified session has been invalidated for some reason.
[hls,applehttp @ 0x145e620] HLS request for url 'https://r1---sn-gwpa-gq2l.googlevideo.com/videoplayback/id/Cx1uJUVCLto.1/itag/93/source/yt_live_broadcast/sq/17510/file/seg.ts?requiressl=yes&ratebypass=yes&cmbypass=yes&goi=160&sgoap=gir=yes;itag=140&sgovp=gir=yes;itag=134&hls_chunk_host=r1---sn-gwpa-gq2l.googlevideo.com&ei=KuxWWdW8NsaNoAP8rK1A&gcr=in&playlist_type=DVR&mm=32&mn=sn-gwpa-gq2l&ms=lv&mv=m&pl=22&mt=1498868718&ip=47.15.9.230&ipbits=0&expire=1498890377&sparams=ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,ei,gcr,playlist_type,mm,mn,ms,mv,pl&signature=96E2DB8417DEF3A420ECC42CF17CCD688B310FDD.696630EB61AFCFCE3968BA1A1FC8397AC11A8668&key=dg_yt0&playlist=index.m3u8&live=1&goap=clen=80970;lmt=1498868778268654&govp=clen=57077;lmt=1498868778268654&dur=5.000', offset 0, playlist 0
[h264 @ 0x1bd2ae0] Current profile doesn't provide more RBSP data in PPS, skipping
Input #0, hls,applehttp, from 'https://manifest.googlevideo.com/api/manifest/hls_playlist/id/Cx1uJUVCLto.1/itag/93/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D134/hls_chunk_host/r1---sn-gwpa-gq2l.googlevideo.com/ei/KuxWWdW8NsaNoAP8rK1A/gcr/in/playlist_type/DVR/mm/32/mn/sn-gwpa-gq2l/ms/lv/mv/m/pl/22/dover/6/mt/1498868718/ip/47.15.9.230/ipbits/0/expire/1498890377/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,ei,gcr,playlist_type,mm,mn,ms,mv,pl/signature/96E2DB8417DEF3A420ECC42CF17CCD688B310FDD.696630EB61AFCFCE3968BA1A1FC8397AC11A8668/key/dg_yt0/playlist/index.m3u8':
Duration: N/A, start: 87237.665333, bitrate: N/A
Program 0
Metadata:
variant_bitrate : 0
Stream #0:0: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 127 kb/s
Stream #0:1: Video: h264 (Main), 1 reference frame ([27][0][0][0] / 0x001B), yuv420p(tv, bt709), 640x360 (640x368) [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 90k tbn, 60 tbc
[mp4 @ 0x1bd52e0] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 @ 0x1bd52e0] Codec for stream 1 does not use global headers but container format requires global headers
Output #0, mp4, to 'file:Pokemon Sun and Moon Episode 32 English Sub-Cx1uJUVCLto.mp4.part':
Metadata:
encoder : Lavf56.40.101
Stream #0:0: Video: h264, 1 reference frame ([33][0][0][0] / 0x0021), yuv420p, 640x360 (0x0) [SAR 1:1 DAR 16:9], q=2-31, 30 fps, 30 tbr, 90k tbn, 90k tbc
Stream #0:1: Audio: aac ([64][0][0][0] / 0x0040), 48000 Hz, stereo, 127 kb/s
Stream mapping:
Stream #0:1 -> #0:0 (copy)
Stream #0:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
[mp4 @ 0x1bd52e0] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
av_interleaved_write_frame(): Operation not permitted
No more output streams to write to, finishing.
[mp4 @ 0x1bd52e0] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
frame= 1 fps=0.0 q=-1.0 Lsize= 8kB time=00:00:00.04 bitrate=1621.1kbits/s
video:8kB audio:1kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.307419%
Input file #0 (https://manifest.googlevideo.com/api/manifest/hls_playlist/id/Cx1uJUVCLto.1/itag/93/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D134/hls_chunk_host/r1---sn-gwpa-gq2l.googlevideo.com/ei/KuxWWdW8NsaNoAP8rK1A/gcr/in/playlist_type/DVR/mm/32/mn/sn-gwpa-gq2l/ms/lv/mv/m/pl/22/dover/6/mt/1498868718/ip/47.15.9.230/ipbits/0/expire/1498890377/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,ei,gcr,playlist_type,mm,mn,ms,mv,pl/signature/96E2DB8417DEF3A420ECC42CF17CCD688B310FDD.696630EB61AFCFCE3968BA1A1FC8397AC11A8668/key/dg_yt0/playlist/index.m3u8):
Input stream #0:0 (audio): 2 packets read (605 bytes);
Input stream #0:1 (video): 1 packets read (7846 bytes);
Total: 3 packets (8451 bytes) demuxed
Output file #0 (file:Pokemon Sun and Moon Episode 32 English Sub-Cx1uJUVCLto.mp4.part):
Output stream #0:0 (video): 1 packets muxed (7846 bytes);
Output stream #0:1 (audio): 2 packets muxed (605 bytes);
Total: 3 packets (8451 bytes) muxed
Conversion failed!
ERROR: ffmpeg exited with code 1
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/bin/youtube-dl/main.py", line 19, in
youtube_dl.main()
File "/usr/local/bin/youtube-dl/youtube_dl/init.py", line 465, in main
_real_main(argv)
File "/usr/local/bin/youtube-dl/youtube_dl/init.py", line 455, in _real_main
retcode = ydl.download(all_urls)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1927, in download
url, force_generic_extractor=self.params.get('force_generic_extractor', False))
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 773, in extract_info
return self.process_ie_result(ie_result, download, extra_info)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 827, in process_ie_result
return self.process_video_result(ie_result, download=download)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1570, in process_video_result
self.process_info(new_info)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1834, in process_info
success = dl(filename, info_dict)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1776, in dl
return fd.download(name, info)
File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 361, in download
return self.real_download(filename, info_dict)
File "/usr/local/bin/youtube-dl/youtube_dl/downloader/external.py", line 57, in real_download
self.get_basename(), retval))
File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 163, in report_error
self.ydl.report_error(*args, **kargs)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 602, in report_error
self.trouble(error_message, tb)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 564, in trouble
tb_data = traceback.format_list(traceback.extract_stack())