[Broken] YouTube: a very small number of videos consistently fail to download #3

jbruchon · 2021-01-06T21:25:32Z

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-f', '137+251', '--cookies', 'Y:\\cookies.txt', '--add-metadata', '--write-description', '--write-info-json', '--write-thumbnail', '--download-archive', 'Y:\\ytdl_archive.txt', '--all-subs', '-ciw', '-o', '%(title)s.%(ext)s', '-v', 'https://www.youtube.com/watch?v=LIdZ2oPyB1Y']
[debug] Loading archive file 'Y:\\ytdl_archive.txt'
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dlc version 2020.11.11-2
[debug] Python version 3.9.0 (CPython) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg git-2020-04-15-51db0a4, ffprobe 4.3.1-2020-10-01-full_build-www.gyan.dev
[debug] Proxy map: {}
[youtube] LIdZ2oPyB1Y: Downloading webpage
[info] Video description is already present
[info] Video subtitle en.vtt is already present
[info] Video subtitle tr.vtt is already present
[info] Video description metadata is already present
[youtube] LIdZ2oPyB1Y: Thumbnail is already present
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[debug] Invoking downloader on 'https://r5---sn-hp57kn7e.googlevideo.com/videoplayback?expire=1606634316&ei=6_bCX8bLNuuDzLUP5pi8oAM&ip=77.81.142.124&id=o-ABxbQ79-SQUNxEEobMptDUyFmzaAcyduZOh0portJKj-&itag=137&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&source=youtube&requiressl=yes&mh=wM&mm=31%2C29&mn=sn-hp57kn7e%2Csn-hp57yne7&ms=au%2Crdu&mv=m&mvi=5&pl=24&initcwndbps=633750&vprv=1&mime=video%2Fmp4&ns=KsUGVusZhmzwLjUH87Cn8yEF&gir=yes&clen=249849935&dur=1342.720&lmt=1572754520057451&mt=1606612362&fvip=5&keepalive=yes&c=WEB&txp=1306222&n=eRfKKH3XA4B5i7zHf36&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgCkzbEg6HpwujHV6Zu9gFcCZ9rPEBjcJ3JK-iZWLrQVYCIQCZoM-u9Slf2GXAjgqhOhsTM0H-fcEx2C6NpT-Ng1sUxQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIgZH75_E7jj_TWqgwOwjw5wGYRqZiq-9THO7z7BrBJHGsCIQCS3s_ppZN__VE--XDQFihwQ5A_saofyzFdOcPNBlEdBQ%3D%3D&ratebypass=yes'
ERROR: unable to download video data: <urlopen error [WinError 10051] A socket operation was attempted to an unreachable network>
Traceback (most recent call last):
  File "urllib\request.py", line 1342, in do_open
  File "http\client.py", line 1255, in request
  File "http\client.py", line 1301, in _send_request
  File "http\client.py", line 1250, in endheaders
  File "http\client.py", line 1010, in _send_output
  File "http\client.py", line 950, in send
  File "http\client.py", line 1417, in connect
  File "http\client.py", line 921, in connect
  File "socket.py", line 843, in create_connection
  File "socket.py", line 831, in create_connection
OSError: [WinError 10051] A socket operation was attempted to an unreachable network

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "youtube_dlc\YoutubeDL.py", line 2014, in process_info
  File "youtube_dlc\YoutubeDL.py", line 1875, in dl
  File "youtube_dlc\downloader\common.py", line 375, in download
  File "youtube_dlc\downloader\http.py", line 349, in real_download
  File "youtube_dlc\downloader\http.py", line 114, in establish_connection
  File "youtube_dlc\downloader\http.py", line 110, in establish_connection
  File "youtube_dlc\YoutubeDL.py", line 2325, in urlopen
  File "urllib\request.py", line 517, in open
  File "urllib\request.py", line 534, in _open
  File "urllib\request.py", line 494, in _call_chain
  File "youtube_dlc\utils.py", line 2736, in https_open
  File "urllib\request.py", line 1345, in do_open
urllib.error.URLError: <urlopen error [WinError 10051] A socket operation was attempted to an unreachable network>

Description

When I archive whole channels, some videos may fail to download due to copyright claims, but there are a few outliers like this one that work fine on YouTube and fail in yt-dlc. This might happen to one out of every 2000 videos I archive, but these same videos are failing the same way for multiple weeks across reboots and new program versions. I noticed that Waterfox seems to be using different servers than yt-dlc for some reason; it seems that the servers yt-dlc is picking up don't actually work. Here's a sample of the network traffic dev console panel for that video:

Also, there does appear to be one mention of the server yt-dlc is trying in the list; it shows 0 bytes transferred.

The text was updated successfully, but these errors were encountered:

pukkandan · 2021-01-07T11:57:26Z

I have never encountered this problem and am unable to reproduce the issue even with the URL you provided. If you know how the issue can be replicated consistently, please let me know

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.youtube.com/watch?v=LIdZ2oPyB1Y', '-f', '137+251', '-v']
[debug] Loading archive file None
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dlc version 2021.01.05-2
[debug] Git HEAD: 5d0c53714
[debug] Python version 3.8.4 (CPython) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg 4.3.1-2020-11-19-full_build-www.gyan.dev, ffprobe 4.3.1-2020-11-19-full_build-www.gyan.dev
[debug] Proxy map: {}
[youtube] LIdZ2oPyB1Y: Downloading webpage
[debug] Sort order given by user:
[debug] Formats sorted by: has_video, quality, extractor_preference, language_preference, dimension, fps, vcodec, acodec, filesize, filesize_approx, tbr, vbr, abr, asr, protocol, vext, aext, has_audio, source_preference, format_id
[info] Downloading format(s) 137+251
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[debug] Invoking downloader on 'https://r8---sn-gwpa-jv3s.googlevideo.com/videoplayback?expire=1610042097&ei=kfb2X5eILpaZz7sP1puO4Aw&ip=2405%3A201%3Af003%3Abc46%3Ab547%3Aea16%3Ad349%3A3686&id=o-AOd1Ic5FAbjV8_EZ5hDDoXhOPcYVk2CCKF9SktNpTMcb&itag=137&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&source=youtube&requiressl=yes&mh=wM&mm=31%2C29&mn=sn-gwpa-jv3s%2Csn-h557snsl&ms=au%2Crdu&mv=m&mvi=8&pcm2cms=yes&pl=38&initcwndbps=235000&vprv=1&mime=video%2Fmp4&ns=_tkH8y3mbM3AONLbZ1S0pz4F&gir=yes&clen=249849935&dur=1342.720&lmt=1572754520057451&mt=1610020111&fvip=5&keepalive=yes&c=WEB&txp=1306222&n=2W9MMO2a6gvr2qSZ&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAPZuFyNkelIIAHWEC4ep90oXsManSC_NjWm7rrM91KPmAiEA61vFWi-v-HkmoBWDgssjscInbdIJwZEfWKsZ1_V-hxg%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpcm2cms%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAPFU0w5raiq49IV6mG8b1kzS4F5stXLQBsdZeZ44JiKAAiEAxulwoqh-KGqp6QkKadmk9oXCIdF6BphQykjJI31P6EE%3D&ratebypass=yes'
[download] Resuming download at byte 31611141
[download] Destination: YouTube BANNING Hacking Videos - Hot Take [LIdZ2oPyB1Y].f137.mp4
[download]  14.3% of 238.28MiB at  2.61MiB/s ETA 01:18
ERROR: Interrupted by user

jbruchon · 2021-01-07T19:49:18Z

YouTube provides more than one host to fetch video data but youtube-dl just rolls with the first one in the list and aborts as seen if it can't reach the host. The program should do what the browser does: retry the fetch with the next host. However, this command is now working for me as well (it didn't work when I originally filed a bug report on the old project). I also run into videos that don't download because the host just pumps out 10x 404s in a row on the first fragment and the program gives up, and those videos also work in the browser, so it may be a similar issue.

Perhaps this should be converted into a feature request to try different hosts if the first one fails? Regardless, feel free to close for now.

pukkandan · 2021-01-07T20:06:37Z

Since the core issue has not been resolved, I will keep this open, but dont expect a fix anytime soon

pukkandan · 2021-03-11T08:32:30Z

@jbruchon Have you experienced this issue since? It seems like it was a problem from youtube's side and doesn't happen anymore

jbruchon · 2021-03-11T14:55:12Z

Nah, I don't think so.

pukkandan · 2021-04-29T00:05:44Z

Reopening since it seems like the issue still exists #283

jbruchon · 2021-04-29T00:43:54Z

It's good to know I'm not crazy after all 💯

Ashish0804 · 2021-05-15T05:17:22Z

something similar happened with this url

(base) ashish@pop-os:~/HDD$ yt-dlp -v https://www.youtube.com/watch?v=JBjHcPPPSo0
[debug] Command-line config: ['-v', 'https://www.youtube.com/watch?v=JBjHcPPPSo0']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] yt-dlp version 2021.05.11 
[debug] Python version 3.7.7 (CPython 64bit) - Linux-5.11.0-7614-generic-x86_64-with-debian-bullseye-sid
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] JBjHcPPPSo0: Downloading webpage
[youtube] JBjHcPPPSo0: Downloading MPD manifest
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] JBjHcPPPSo0: Downloading format(s) 248+251
[debug] Invoking downloader on 'https://manifest.googlevideo.com/api/manifest/dash/expire/1621037087/ei/v7ueYKS9BKmmz7sP76SJ-AE/ip/122.162.250.79/id/2418c770f3cf4a8d/source/youtube/requiressl/yes/playback_host/r2---sn-ci5gup-qxad.googlevideo.com/mh/17/mm/31%2C26/mn/sn-ci5gup-qxad%2Csn-cvh7knes/ms/au%2Conr/mv/m/mvi/2/pl/21/tx/23964905/txs/23964905%2C23964906%2C24034796%2C24034797%2C24034798/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/initcwndbps/1025000/vprv/1/mt/1621015000/fvip/2/keepalive/yes/fexp/24001373%2C24007246/itag/0/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Ctx%2Ctxs%2Chfr%2Cas%2Cvprv%2Citag/sig/AOq0QJ8wRQIgN8Ob8wiRlswcdMGietA3g7uH9sw7B5XMHMwe5ACelWwCIQC6oGoPiMKGsMZqLhd8FAQ5BcKxKT5uOyLBPjQKEpj0ng%3D%3D/lsparams/playback_host%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps/lsig/AG3C_xAwRQIgVFAs1BzMV8i5T771WVFo4-zSOimv0dmJ7iv8QFCs47MCIQD4xshVoryRllQy6SenTziKyEDMY4zg624wAyheAjQkcQ%3D%3D'
[dashsegments] Total fragments: 21
[download] Destination: Gypsy Crusader - Got Bullied for Being Gay (FULL VIDEO) {2} _ VPE [JBjHcPPPSo0].f248.webm
ERROR: unable to download video data: <urlopen error [Errno 101] Network is unreachable>
Traceback (most recent call last):
  File "/home/ashish/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/home/ashish/miniconda3/lib/python3.7/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/ashish/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/ashish/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/ashish/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/home/ashish/miniconda3/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/home/ashish/miniconda3/lib/python3.7/http/client.py", line 1414, in connect
    super().connect()
  File "/home/ashish/miniconda3/lib/python3.7/http/client.py", line 938, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/home/ashish/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
    raise err
  File "/home/ashish/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ashish/miniconda3/lib/python3.7/site-packages/yt_dlp/YoutubeDL.py", line 2562, in process_info
    partial_success, real_download = self.dl(fname, new_info)
  File "/home/ashish/miniconda3/lib/python3.7/site-packages/yt_dlp/YoutubeDL.py", line 2257, in dl
    return fd.download(name, new_info, subtitle)
  File "/home/ashish/miniconda3/lib/python3.7/site-packages/yt_dlp/downloader/common.py", line 381, in download
    return self.real_download(filename, info_dict), True
  File "/home/ashish/miniconda3/lib/python3.7/site-packages/yt_dlp/downloader/dash.py", line 180, in real_download
    frag_content, frag_index = download_fragment(fragment)
  File "/home/ashish/miniconda3/lib/python3.7/site-packages/yt_dlp/downloader/dash.py", line 92, in download_fragment
    success, frag_content = self._download_fragment(ctx, fragment_url, info_dict)
  File "/home/ashish/miniconda3/lib/python3.7/site-packages/yt_dlp/downloader/fragment.py", line 110, in _download_fragment
    success = ctx['dl'].download(fragment_filename, fragment_info_dict)
  File "/home/ashish/miniconda3/lib/python3.7/site-packages/yt_dlp/downloader/common.py", line 381, in download
    return self.real_download(filename, info_dict), True
  File "/home/ashish/miniconda3/lib/python3.7/site-packages/yt_dlp/downloader/http.py", line 352, in real_download
    establish_connection()
  File "/home/ashish/miniconda3/lib/python3.7/site-packages/yt_dlp/downloader/http.py", line 117, in establish_connection
    raise err
  File "/home/ashish/miniconda3/lib/python3.7/site-packages/yt_dlp/downloader/http.py", line 111, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "/home/ashish/miniconda3/lib/python3.7/site-packages/yt_dlp/YoutubeDL.py", line 2994, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/home/ashish/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/home/ashish/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/home/ashish/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/home/ashish/miniconda3/lib/python3.7/site-packages/yt_dlp/utils.py", line 2794, in https_open
    req, **kwargs)
  File "/home/ashish/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 101] Network is unreachable>

that being said, i am able to download it now.

But this is a legitimate issue it seems.

3zhang · 2021-09-09T03:00:18Z

Similar problem. But after I updated the cookies it downloaded successfully.

pukkandan · 2021-12-13T13:13:45Z

Closed in favor of #1986

Updates from bashonly: * sort class above function * for livestreams, `livestream` is `True` in catalogue Co-authored-by: bashonly <88596187+bashonly@users.noreply.github.com>

quackitiy · 2024-03-21T19:36:06Z

im also having this same issue on every video "A socket operation was attempted to an unreachable network." D:
the thumbnail, infojson, description, all download fine, except the videos. like one of the other people said here with updating cookies, i deleted cookies.txt, ran it, and it fixed :)

pukkandan added the cant-reproduce The issue cannot be reliably reproduced label Jan 7, 2021

pukkandan added the help-wanted Extra attention is needed label Feb 1, 2021

jbruchon closed this as completed Mar 11, 2021

pukkandan reopened this Apr 29, 2021

pukkandan mentioned this issue Apr 29, 2021

[Broken] YouTube - Fallback to a different googlevideo host #283

Closed

6 tasks

pukkandan removed the cant-reproduce The issue cannot be reliably reproduced label Apr 29, 2021

pukkandan mentioned this issue Oct 16, 2021

[Broken] youtube: urlopen error EOF occurred in violation of protocol (_ssl.c:1131) #1307

Closed

7 tasks

VVfearless mentioned this issue Oct 19, 2021

[Question] yt-dlp not worked on macOS. #1339

Closed

5 tasks

pukkandan mentioned this issue Dec 13, 2021

[youtube] yt-dlp fails to fall back if primary video server is unavailable #1986

Open

7 tasks

pukkandan closed this as completed Dec 13, 2021

pukkandan added the duplicate This issue or pull request already exists label Dec 13, 2021

ralioc mentioned this issue Jan 15, 2022

Unable to download API page: IncompleteRead #2344

Closed

7 tasks

JCU0 mentioned this issue Mar 13, 2022

Option to retry when connection is unstable: [Errno 101] Network is unreachable #3056

Closed

6 tasks

svenop5 mentioned this issue Jun 2, 2022

[teachable] ERROR: Unable to find Video URL https://edu.ecomsuccess.pk/courses/enrolled/903296 #3564

Open

7 tasks

Ulmo mentioned this issue Nov 3, 2022

Odysee members-only videos using cookies results in "authentication required" #5431

Closed

10 tasks

marlouz mentioned this issue Apr 29, 2023

Error downloading from OpenRec #6946

Closed

10 tasks

user7230724 mentioned this issue Apr 29, 2024

[YouTube] Unable to download video #9818

Closed

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Broken] YouTube: a very small number of videos consistently fail to download #3

[Broken] YouTube: a very small number of videos consistently fail to download #3

jbruchon commented Jan 6, 2021

pukkandan commented Jan 7, 2021

jbruchon commented Jan 7, 2021

pukkandan commented Jan 7, 2021 •

edited

pukkandan commented Mar 11, 2021

jbruchon commented Mar 11, 2021

pukkandan commented Apr 29, 2021

jbruchon commented Apr 29, 2021

Ashish0804 commented May 15, 2021

3zhang commented Sep 9, 2021

pukkandan commented Dec 13, 2021

quackitiy commented Mar 21, 2024 •

edited

[Broken] YouTube: a very small number of videos consistently fail to download #3

[Broken] YouTube: a very small number of videos consistently fail to download #3

Comments

jbruchon commented Jan 6, 2021

Verbose log

Description

pukkandan commented Jan 7, 2021

jbruchon commented Jan 7, 2021

pukkandan commented Jan 7, 2021 • edited

pukkandan commented Mar 11, 2021

jbruchon commented Mar 11, 2021

pukkandan commented Apr 29, 2021

jbruchon commented Apr 29, 2021

Ashish0804 commented May 15, 2021

3zhang commented Sep 9, 2021

pukkandan commented Dec 13, 2021

quackitiy commented Mar 21, 2024 • edited

pukkandan commented Jan 7, 2021 •

edited

quackitiy commented Mar 21, 2024 •

edited