[youtube] yt-dlp fails to fall back if primary video server is unavailable #1986
Comments
|
Duplicate of #3, but this issue report is much more well described, so I will close that instead
yt-dlp has a default |
pukkandan
added
enhancement
|
I'm not very experienced with ytdl code, so this is extremely hacky, but I've been able to implement a poor proof of concept. It works for downloading videos when yt-dlp is used standalone, but does not work when used via mpv's ytdl_hook. index 6290884..f4981a8 100644
--- a/yt_dlp/downloader/http.py
+++ b/yt_dlp/downloader/http.py
@@ -374,6 +374,17 @@ class HttpFD(FileDownloader):
except RetryDownload as e:
count += 1
if count <= retries:
+ fallback_m = re.search(r'(?:---|\.)([^.]+)\.googlevideo\.com/.*&mn=([^&]*)', url)
+ if fallback_m:
+ host = fallback_m.group(1)
+ fallbacks = fallback_m.group(2).split("%2C")
+ assert host in fallbacks
+ newhost = fallbacks[(fallbacks.index(host) + 1) % len(fallbacks)]
+ url = re.sub(host+r"\.googlevideo\.com",newhost+".googlevideo.com",url)
+ if "fallback_count=" in url:
+ url = re.sub(r"fallback_count=[^&]*","fallback_count={}".format(count),url)
+ else:
+ url += "&fallback_count={}".format(count)
self.report_retry(e.source_error, count, retries)
else:
self.to_screen(f'[download] Got server HTTP error: {e.source_error}')This is obviously a horrible solution, as it places extractor-specific code deep in generic http download code, and it doesn't always work as the broken ytdl_hook shows. |
|
Looks reasonable as a POC. ofc, like you said, this is not a proper solution and won't be merged as-is. Maybe the extractor can add a
mpv will have to implement it separately since they only use our extractors, not our downloaders. If we go with the |
|
We really need this fix. I experienced about a dozen of such videos with broken primary google server link, but works fine with altenative servers provided via |
|
To anyone who wants to implement this: Seems like there are actually two kinds of URL formats youtube uses, one of them I believe is used for DASH (live videos, premieres?) and uses parameters between slashes in the URL path instead of using the URL query part. --- a/yt_dlp/downloader/http.py
+++ b/yt_dlp/downloader/http.py
@@ -374,6 +374,15 @@ class HttpFD(FileDownloader):
except RetryDownload as e:
count += 1
if count <= retries:
+ fallback_m = re.search(r'(?:---|\.)([^./]+)\.googlevideo\.com/.*[&/]mn[=/]([^/&]*)', url)
+ if fallback_m:
+ host = fallback_m.group(1)
+ fallbacks = fallback_m.group(2).split("%2C")
+ assert host in fallbacks
+ newhost = fallbacks[(fallbacks.index(host) + 1) % len(fallbacks)]
+ url = re.sub(host+r"\.googlevideo\.com",newhost+".googlevideo.com",url)
+ for prop in ["manifest_url","fragment_base_url"]:
+ if prop in f: f[prop] = re.sub(host+r"\.googlevideo\.com",newhost+".googlevideo.com",f[prop])
self.report_retry(e.source_error, count, retries)
else:
self.to_screen(f'[download] Got server HTTP error: {e.source_error}') |
|
Additionally, here's a patch I'm using to block the specific server that doesn't work for me, might work as inspiration for where to place the fallback urls in the extractor code, but making that actually work would require redesigning some ytdl internals I'm not entirely comfortable with. --- a/yt_dlp/downloader/http.py
+++ b/yt_dlp/downloader/http.py
@@ -369,6 +369,8 @@ class HttpFD(FileDownloader):
while count <= retries:
try:
+ if "sn-u2oxu-3g3s.googlevideo.com/" in url:
+ raise RetryDownload(RuntimeError("Bad video server"))
establish_connection()
return download()
except RetryDownload as e:
--- a/yt_dlp/extractor/youtube.py
+++ b/yt_dlp/extractor/youtube.py
@@ -2846,6 +2846,21 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
# When throttling issue is fully fixed, remove this
self._sort_formats(formats, ('quality', 'res', 'fps', 'hdr:12', 'source', 'codec:vp9.2', 'lang', 'proto'))
+ for f in formats:
+ if "sn-u2oxu-3g3s.googlevideo.com/" in f.get("url"):
+ url = f["url"]
+ fallback_m = re.search(r'(?:---|\.)([^./]+)\.googlevideo\.com/.*[&/]mn[=/]([^/&]*)', url)
+ #print(url + "\n", fallback_m, fallback_m.group(2) if fallback_m else None)
+ assert fallback_m
+ host = fallback_m.group(1)
+ fallbacks = fallback_m.group(2).split("%2C")
+ assert host in fallbacks
+ newhost = fallbacks[(fallbacks.index(host) + 1) % len(fallbacks)]
+ url = re.sub(host+r"\.googlevideo\.com",newhost+".googlevideo.com",url)
+ f["url"] = url
+ for prop in ["manifest_url","fragment_base_url"]:
+ if prop in f: f[prop] = re.sub(host+r"\.googlevideo\.com",newhost+".googlevideo.com",f[prop])
+
info = {
'id': video_id,
'title': self._live_title(video_title) if is_live else video_title, |
|
Thanks @mateon1 for the ipv6 note. Indeed turning ipv6 off hotfixed the problem for me 🙂 So on linux it just needed those 2 commands with no reboot required |
You can also just force yt-dlp to use ip4 with |
Ah wopsi I didn't see that. That is ofc way better 👍 |
|
Has the fallback mechanism been implemented in any way yet? Is there some kind of flag one can use? |
|
You can try out #5639 or #1986 (comment), otherwise no |
|
The PR has conflicts. Maybe the patch will work. If I get tired of opening a separate, clean browser instance for every single youtube video, I might give it a shot. Thanks. |
|
Have there been any updates in terms of implementing a fallback method when the primary server is not reachable? I've been getting the same error since yesterday with yt-dlp for any youtube video I try (with either Or is it maybe possible to override using a different server via cli/url parameters as a workaround? edit: Turns out I just had an unlucky video selection, some videos still load as usual, others fail with the described issue |
Checklist
Region
No response
Description
For whatever reason, the server
r1---sn-u2oxu-3g3s.googlevideo.comis unreachable for me. Despite this, videos with this server as the primary video server continue to play in the browser.Youtube provides a fallback server in the
mnquery field of the/videoplaybacklink, and the browser uses that after the first connection fails or times out.After failing the request:
the browser connects to
which succeeds.
yt-dlp should use that server as a fallback if connection to the primary one fails before it gives up.
NOTE:
I believe the debug log fails with 'Network is unreachable' because it tries to connect over broken ipv6 after the ipv4 connection fails.
NOTE 2:
In the browser, the first connection times out significantly quicker than in yt-dlp. In Firefox devtools, the first connection to
sn-u2oxu-3g3sgets queued at 1.43s, and the connection tosn-u2oxu-f5fedstarts at 2.50s.yt-dlp with the
-4flag takes so long to retry all the connections 10 times that I've managed to capture a Firefox network trace of the relevant youtube page and write most of this report while waiting for the command to finally finish.Verbose log
The text was updated successfully, but these errors were encountered: