zlib.error: Error -5 while decompressing data: incomplete or truncated stream

[remitamine]

youtube-dl -v http://www.skysports.com/watch/video/10018211/rodgers-sacked-by-liverpool
[debug] System config: []
[debug] User config: ['--external-downloader', 'aria2c', '--external-downloader-args', '-x 16', '-f', '[height<=?720]', '--sub-lang', 'ar,arME,en', '--write-sub', '--no-check-certificate']
[debug] Command-line args: ['-v', 'http://www.skysports.com/watch/video/10018211/rodgers-sacked-by-liverpool']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.09.28
[debug] Python version 3.5.0 - Linux-4.2.3-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 2.8.1, ffprobe 2.8.1, rtmpdump 2.4
[debug] Proxy map: {}
[generic] rodgers-sacked-by-liverpool: Requesting header
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/youtube_dl/utils.py", line 674, in deflate
    return zlib.decompress(data, -zlib.MAX_WBITS)
zlib.error: Error -5 while decompressing data: incomplete or truncated stream

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/youtube-dl", line 9, in <module>
    load_entry_point('youtube-dl==2015.9.28', 'console_scripts', 'youtube-dl')()
  File "/usr/lib/python3.5/site-packages/youtube_dl/__init__.py", line 410, in main
    _real_main(argv)
  File "/usr/lib/python3.5/site-packages/youtube_dl/__init__.py", line 400, in _real_main
    retcode = ydl.download(all_urls)
  File "/usr/lib/python3.5/site-packages/youtube_dl/YoutubeDL.py", line 1658, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/usr/lib/python3.5/site-packages/youtube_dl/YoutubeDL.py", line 660, in extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python3.5/site-packages/youtube_dl/extractor/common.py", line 288, in extract
    return self._real_extract(url)
  File "/usr/lib/python3.5/site-packages/youtube_dl/extractor/generic.py", line 1162, in _real_extract
    fatal=False)
  File "/usr/lib/python3.5/site-packages/youtube_dl/extractor/common.py", line 327, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/lib/python3.5/site-packages/youtube_dl/YoutubeDL.py", line 1865, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 471, in open
    response = meth(req, response)
  File "/usr/lib/python3.5/site-packages/youtube_dl/utils.py", line 748, in http_response
    gz = io.BytesIO(self.deflate(resp.read()))
  File "/usr/lib/python3.5/site-packages/youtube_dl/utils.py", line 676, in deflate
    return zlib.decompress(data)
zlib.error: Error -5 while decompressing data: incomplete or truncated stream

[jaimeMF]

(For reference, this uses the url posted in #7066)

This only happens for HEAD requests because it returns no data, therefore zlib can't decompress it. This simple patch "fixes" it:

diff --git a/youtube_dl/utils.py b/youtube_dl/utils.py
index 7dbe256..dee3b00 100644
--- a/youtube_dl/utils.py
+++ b/youtube_dl/utils.py
@@ -670,6 +670,9 @@ class YoutubeDLHandler(compat_urllib_request.HTTPHandler):

     @staticmethod
     def deflate(data):
+        # HEAD requests produce no data, which is not a valid compressed file
+        if not data:
+            return data
         try:
             return zlib.decompress(data, -zlib.MAX_WBITS)
         except zlib.error:

Although it could be better to handle it in YoutubeDLHandler.http_response.

[nielsangho]

How do I apply this "fix"? @jaimeMF

[ole]

@jaimeMF It's been five years, but thank you for this patch. I was just able to fix a failing download by applying your patch to youtube-dl version 2020.11.21.1.

The URL of the failing page is https://www.pointfree.co/episodes/ep100-a-tour-of-the-composable-architecture-part-1

Without the patch:

$ youtube-dl --version
2020.11.21.1

$ youtube-dl -v https://www.pointfree.co/episodes/ep100-a-tour-of-the-composable-architecture-part-1
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.pointfree.co/episodes/ep100-a-tour-of-the-composable-architecture-part-1']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.11.21.1
[debug] Python version 3.9.0 (CPython) - macOS-10.15.7-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.3.1, ffprobe 4.3.1, rtmpdump 2.4
[debug] Proxy map: {}
[generic] ep100-a-tour-of-the-composable-architecture-part-1: Requesting header
Traceback (most recent call last):
  File "/usr/local/Cellar/youtube-dl/2020.11.21.1_1/libexec/lib/python3.9/site-packages/youtube_dl/utils.py", line 2588, in deflate
    return zlib.decompress(data, -zlib.MAX_WBITS)
zlib.error: Error -5 while decompressing data: incomplete or truncated stream

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl", line 33, in <module>
    sys.exit(load_entry_point('youtube-dl==2020.11.21.1', 'console_scripts', 'youtube-dl')())
  File "/usr/local/Cellar/youtube-dl/2020.11.21.1_1/libexec/lib/python3.9/site-packages/youtube_dl/__init__.py", line 474, in main
    _real_main(argv)
  File "/usr/local/Cellar/youtube-dl/2020.11.21.1_1/libexec/lib/python3.9/site-packages/youtube_dl/__init__.py", line 464, in _real_main
    retcode = ydl.download(all_urls)
  File "/usr/local/Cellar/youtube-dl/2020.11.21.1_1/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 2028, in download
    res = self.extract_info(
  File "/usr/local/Cellar/youtube-dl/2020.11.21.1_1/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 796, in extract_info
    return self.__extract_info(url, ie, download, extra_info, process)
  File "/usr/local/Cellar/youtube-dl/2020.11.21.1_1/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 803, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/Cellar/youtube-dl/2020.11.21.1_1/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 824, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/Cellar/youtube-dl/2020.11.21.1_1/libexec/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 532, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/Cellar/youtube-dl/2020.11.21.1_1/libexec/lib/python3.9/site-packages/youtube_dl/extractor/generic.py", line 2296, in _real_extract
    head_response = self._request_webpage(
  File "/usr/local/Cellar/youtube-dl/2020.11.21.1_1/libexec/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 632, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/Cellar/youtube-dl/2020.11.21.1_1/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 2248, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/Cellar/python@3.9/3.9.0_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/local/Cellar/youtube-dl/2020.11.21.1_1/libexec/lib/python3.9/site-packages/youtube_dl/utils.py", line 2648, in http_response
    gz = io.BytesIO(self.deflate(resp.read()))
  File "/usr/local/Cellar/youtube-dl/2020.11.21.1_1/libexec/lib/python3.9/site-packages/youtube_dl/utils.py", line 2590, in deflate
    return zlib.decompress(data)
zlib.error: Error -5 while decompressing data: incomplete or truncated stream

With the patch:

youtube-dl https://www.pointfree.co/episodes/ep100-a-tour-of-the-composable-architecture-part-1
[generic] ep100-a-tour-of-the-composable-architecture-part-1: Requesting header
WARNING: Could not send HEAD request to https://www.pointfree.co/episodes/ep100-a-tour-of-the-composable-architecture-part-1: HTTP Error 404: Not Found
[generic] ep100-a-tour-of-the-composable-architecture-part-1: Downloading webpage
WARNING: Falling back on generic information extractor.
[generic] ep100-a-tour-of-the-composable-architecture-part-1: Extracting information
[download] Downloading playlist: Episode #100: A Tour of the Composable Architecture: Part 1
[generic] playlist Episode #100: A Tour of the Composable Architecture: Part 1: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[vimeo] 414016119: Downloading webpage
[vimeo] 414016119: Extracting information
[vimeo] 414016119: Downloading JSON metadata
WARNING: Unable to download JSON metadata: HTTP Error 404: Not Found
[vimeo] 414016119: Downloading fastly_skyfire m3u8 information
[vimeo] 414016119: Downloading fastly_skyfire m3u8 information
[vimeo] 414016119: Downloading fastly_skyfire MPD information
[vimeo] 414016119: Downloading fastly_skyfire MPD information
[dashsegments] Total fragments: 327
[download] Destination: 0100-414016119.fdash-fastly_skyfire_sep-video-b4791714.mp4
[download]   4.7% of ~780.34MiB at  6.73MiB/s ETA 02:02^C

It would be great for this patch or something like it to make it into youtube-dl.