Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Downloading youtube live broadcast using aria2c stops working for a long time #8212

Closed
8 of 9 tasks
sxx520 opened this issue Sep 27, 2023 · 1 comment · Fixed by #8339
Closed
8 of 9 tasks

Downloading youtube live broadcast using aria2c stops working for a long time #8212

sxx520 opened this issue Sep 27, 2023 · 1 comment · Fixed by #8339
Labels
bug Bug that is not site-specific patch-available There is patch available that should fix this issue. Someone needs to make a PR with it

Comments

@sxx520
Copy link

sxx520 commented Sep 27, 2023

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

  • I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

Please make sure the question is worded well enough to be understood

Use aria2c to download youtube live broadcast but no response for more than 10 minutes

Provide verbose output that clearly demonstrates the problem

  • Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
  • If using API, add 'verbose': True to YoutubeDL params instead
  • Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

yt-dlp https://www.youtube.com/watch?v=iZlybDrARYw --live-from-start  --external-downloader aria2c --external-downloader-args '-x 4 -s 4' --proxy http://127.0.0.1:7890/ -v
>>
[debug] Command-line config: ['https://www.youtube.com/watch?v=iZlybDrARYw', '--live-from-start', '--external-downloader', 'aria2c', '--external-downloader-args', '-x 4 -s 4', '--proxy', 'http://127.0.0.1:7890/', '-v']
[debug] Encodings: locale cp936, fs utf-8, pref cp936, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.09.24 [088add956] (pip)
[debug] Python 3.11.5 (CPython AMD64 64bit) - Windows-10-10.0.18363-SP0 (OpenSSL 3.0.9 30 May 2023)
[debug] exe versions: ffmpeg 2023-09-04-git-f8503b4c33-full_build-www.gyan.dev (setts), ffprobe 2023-09-04-git-f8503b4c33-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.07.22, mutagen-1.47.0, sqlite3-3.42.0, websockets-11.0.3
[debug] Proxy map: {'all': 'http://127.0.0.1:7890/'}
WARNING: External Downloader arguments given without specifying name. The arguments will be given to aria2c
[debug] Loaded 1886 extractors
[youtube] Extracting URL: https://www.youtube.com/watch?v=iZlybDrARYw
[youtube] iZlybDrARYw: Downloading webpage
[youtube] iZlybDrARYw: Downloading ios player API JSON
[youtube] iZlybDrARYw: Downloading android player API JSON
[youtube] iZlybDrARYw: Downloading MPD manifest
[youtube] iZlybDrARYw: Downloading MPD manifest
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, size, br, asr, vext, aext, hasaud, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] iZlybDrARYw: Downloading 1 format(s): 248+140
[debug] Invoking dashsegments downloader on "https://manifest.googlevideo.com/api/manifest/dash/expire/1695810646/ei/9q8TZa3rBouQvcAPqsKUyAw/ip/103.148.72.4/id/iZlybDrARYw.1/source/yt_live_broadcast/requiressl/yes/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/spc/UWF9f37Bbs9zIER8dzoMMOC5gyf4tYk/vprv/1/pacing/0/itag_bl/376%2C377%2C384%2C385%2C612%2C613%2C617%2C619%2C623%2C628%2C655%2C656%2C660%2C662%2C666%2C671/keepalive/yes/fexp/24007246/beids/24350018/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Cas%2Cspc%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIgCCAomXbOj5swOzRb5mgLMwoY0hYU2rru0vDNDzDbtrACIQDkwrknQqHdqZcjjtPZ3l54flfmq60LEYZFk0K-4BwKrQ%3D%3D", "https://manifest.googlevideo.com/api/manifest/dash/expire/1695810646/ei/9q8TZa3rBouQvcAPqsKUyAw/ip/103.148.72.4/id/iZlybDrARYw.1/source/yt_live_broadcast/requiressl/yes/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/spc/UWF9f37Bbs9zIER8dzoMMOC5gyf4tYk/vprv/1/pacing/0/itag_bl/376%2C377%2C384%2C385%2C612%2C613%2C617%2C619%2C623%2C628%2C655%2C656%2C660%2C662%2C666%2C671/keepalive/yes/fexp/24007246/beids/24350018/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Cas%2Cspc%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIgCCAomXbOj5swOzRb5mgLMwoY0hYU2rru0vDNDzDbtrACIQDkwrknQqHdqZcjjtPZ3l54flfmq60LEYZFk0K-4BwKrQ%3D%3D"
[dashsegments] Total fragments: unknown (live)
[dashsegments] Fragment downloads will be delegated to aria2c
[debug] [youtube] [iZlybDrARYw] Generating fragments for format 248
@sxx520 sxx520 added the question Question label Sep 27, 2023
@bashonly
Copy link
Member

bashonly commented Sep 27, 2023

yt-dlp is generating the list of fragment URLs that it will pass to aria2c. The problem is, the livestream is ongoing, so it will keep generating fragment URLs until the stream ends. And the already-generated fragment URLs may be expired by the time the stream ends.

Basically, aria2c can't support --live-from-start downloads of ongoing livestreams. It can support downloading post-live streams with --live-from-start, because those have an endpoint, and it won't take minutes/hours/days/forever to generate the fragment URLs. But I don't think it was ever intended that the user should be able to hand off ongoing livestream --live-from-start downloads to aria2c. So this is a bug.

It looks like there was a check added in yt_dlp/downloader/__init__.py for this, but that check is negated later in yt_dlp/downloader/dash.py, where get_suitable_downloader is called again with a protocol of dash_frag_urls.

I think we need to do something like this for a fix:

diff --git a/yt_dlp/downloader/dash.py b/yt_dlp/downloader/dash.py
index 4328d739c..7a66867b6 100644
--- a/yt_dlp/downloader/dash.py
+++ b/yt_dlp/downloader/dash.py
@@ -15,12 +15,15 @@ class DashSegmentsFD(FragmentFD):
     FD_NAME = 'dashsegments'
 
     def real_download(self, filename, info_dict):
-        if info_dict.get('is_live') and set(info_dict['protocol'].split('+')) != {'http_dash_segments_generator'}:
+        if set(info_dict['protocol'].split('+')) == {'http_dash_segments_generator'}:
+            real_downloader = None
+        elif info_dict.get('is_live'):
             self.report_error('Live DASH videos are not supported')
+        else:
+            real_downloader = get_suitable_downloader(
+                info_dict, self.params, None, protocol='dash_frag_urls', to_stdout=(filename == '-'))
 
         real_start = time.time()
-        real_downloader = get_suitable_downloader(
-            info_dict, self.params, None, protocol='dash_frag_urls', to_stdout=(filename == '-'))
 
         requested_formats = [{**info_dict, **fmt} for fmt in info_dict.get('requested_formats', [])]
         args = []

TL;DR: don't use aria2c with ongoing livestream --live-from-start downloads
Use -N if you want concurrent fragment downloads

@bashonly bashonly added bug Bug that is not site-specific patch-available There is patch available that should fix this issue. Someone needs to make a PR with it core-triage triage requested from a core dev and removed question Question labels Sep 27, 2023
@bashonly bashonly removed the core-triage triage requested from a core dev label Nov 7, 2023
bashonly added a commit that referenced this issue Nov 7, 2023
aalsuwaidi pushed a commit to aalsuwaidi/yt-dlp that referenced this issue Apr 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Bug that is not site-specific patch-available There is patch available that should fix this issue. Someone needs to make a PR with it
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants