Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--ignore-errors + DASH keeps going even when segment download fails #10497

Closed
ivan opened this issue Aug 30, 2016 · 14 comments
Closed

--ignore-errors + DASH keeps going even when segment download fails #10497

ivan opened this issue Aug 30, 2016 · 14 comments
Labels
bug

Comments

@ivan
Copy link
Contributor

@ivan ivan commented Aug 30, 2016

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like that [x])
  • Use Preview tab to see how your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2016.08.28. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2016.08.28

Before submitting an issue make sure you have:

  • At least skimmed through README and most notably FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

If you use --ignore-errors, download a DASH format, and a segment fails to download, youtube-dl --ignore-errors will keep going, trying to download every other segment as well:

# PYTHONPATH=. python3 -m youtube_dl -f 135 --ignore-errors -v PJXE91qZP3w
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-f', '135', '--ignore-errors', '-v', 'PJXE91qZP3w']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.08.28
[debug] Git HEAD: da30a20
[debug] Python version 3.5.2 - [...]
[debug] exe versions: none
[debug] Proxy map: {}
[youtube] PJXE91qZP3w: Downloading webpage
[youtube] PJXE91qZP3w: Downloading video info webpage
[youtube] PJXE91qZP3w: Extracting video information
[youtube] PJXE91qZP3w: Downloading MPD manifest
[youtube] PJXE91qZP3w: Downloading MPD manifest
[debug] Invoking downloader on 'https://r5---sn-n4v7sn7z.googlevideo.com/videoplayback/id/3c95c4f75a993f7c/itag/135/source/yt_otf/requiressl/yes/pl/18/mv/m/ms/au/nh/IgpwcjAzLnNqYzA3KgkxMjcuMC4wLjE/mn/sn-n4v7sn7z/mm/31/initcwndbps/400000/ratebypass/yes/mime/video%2Fmp4/otfp/1/otf/1/lmt/1437675588597031/sver/3/signature/65CBE319286895C45C9BD083161B0A752521AD84.48DB63EF63B251C18D1B38411C97712CB537A3E2/mt/1472530870/key/dg_yt0/upn/EAAnQAt53Qw/ip/[REMOVED]/ipbits/0/expire/1472552866/sparams/ip,ipbits,expire,id,itag,source,requiressl,pl,mv,ms,nh,mn,mm,initcwndbps,ratebypass,mime,otfp,otf,lmt/'
[dashsegments] Total fragments: 1037
[download] Destination: 07232015 新聞追追追-PJXE91qZP3w.mp4
[download] Got server HTTP error. Retrying fragment Init (attempt 1 of 10)...
[download] Got server HTTP error. Retrying fragment Init (attempt 2 of 10)...
[download] Got server HTTP error. Retrying fragment Init (attempt 3 of 10)...
[download] Got server HTTP error. Retrying fragment Init (attempt 4 of 10)...
[download] Got server HTTP error. Retrying fragment Init (attempt 5 of 10)...
[download] Got server HTTP error. Retrying fragment Init (attempt 6 of 10)...
[download] Got server HTTP error. Retrying fragment Init (attempt 7 of 10)...
[download] Got server HTTP error. Retrying fragment Init (attempt 8 of 10)...
[download] Got server HTTP error. Retrying fragment Init (attempt 9 of 10)...
[download] Got server HTTP error. Retrying fragment Init (attempt 10 of 10)...
ERROR: giving up after 10 fragment retries
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "youtube-dl/youtube_dl/__main__.py", line 19, in <module>
    youtube_dl.main()
  File "youtube-dl/youtube_dl/__init__.py", line 433, in main
    _real_main(argv)
  File "youtube-dl/youtube_dl/__init__.py", line 423, in _real_main
    retcode = ydl.download(all_urls)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 1788, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 702, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 748, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 1432, in process_video_result
    self.process_info(new_info)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 1696, in process_info
    success = dl(filename, info_dict)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 1638, in dl
    return fd.download(name, info)
  File "youtube-dl/youtube_dl/downloader/common.py", line 354, in download
    return self.real_download(filename, info_dict)
  File "youtube-dl/youtube_dl/downloader/dash.py", line 72, in real_download
    append_url_to_file(initialization_url, ctx['tmpfilename'], 'Init')
  File "youtube-dl/youtube_dl/downloader/dash.py", line 68, in append_url_to_file
    self.report_error('giving up after %s fragment retries' % fragment_retries)
  File "youtube-dl/youtube_dl/downloader/common.py", line 162, in report_error
    self.ydl.report_error(*args, **kargs)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 565, in report_error
    self.trouble(error_message, tb)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 527, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

[download] Got server HTTP error. Retrying fragment Seg0 (attempt 1 of 10)...
[download] Got server HTTP error. Retrying fragment Seg0 (attempt 2 of 10)...
[download] Got server HTTP error. Retrying fragment Seg0 (attempt 3 of 10)...
[download] Got server HTTP error. Retrying fragment Seg0 (attempt 4 of 10)...
[download] Got server HTTP error. Retrying fragment Seg0 (attempt 5 of 10)...
[download] Got server HTTP error. Retrying fragment Seg0 (attempt 6 of 10)...
[download] Got server HTTP error. Retrying fragment Seg0 (attempt 7 of 10)...
[download] Got server HTTP error. Retrying fragment Seg0 (attempt 8 of 10)...
[download] Got server HTTP error. Retrying fragment Seg0 (attempt 9 of 10)...
[download] Got server HTTP error. Retrying fragment Seg0 (attempt 10 of 10)...
ERROR: giving up after 10 fragment retries
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "youtube-dl/youtube_dl/__main__.py", line 19, in <module>
    youtube_dl.main()
  File "youtube-dl/youtube_dl/__init__.py", line 433, in main
    _real_main(argv)
  File "youtube-dl/youtube_dl/__init__.py", line 423, in _real_main
    retcode = ydl.download(all_urls)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 1788, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 702, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 748, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 1432, in process_video_result
    self.process_info(new_info)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 1696, in process_info
    success = dl(filename, info_dict)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 1638, in dl
    return fd.download(name, info)
  File "youtube-dl/youtube_dl/downloader/common.py", line 354, in download
    return self.real_download(filename, info_dict)
  File "youtube-dl/youtube_dl/downloader/dash.py", line 74, in real_download
    append_url_to_file(segment_url, ctx['tmpfilename'], 'Seg%d' % i)
  File "youtube-dl/youtube_dl/downloader/dash.py", line 68, in append_url_to_file
    self.report_error('giving up after %s fragment retries' % fragment_retries)
  File "youtube-dl/youtube_dl/downloader/common.py", line 162, in report_error
    self.ydl.report_error(*args, **kargs)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 565, in report_error
    self.trouble(error_message, tb)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 527, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

[download] Got server HTTP error. Retrying fragment Seg1 (attempt 1 of 10)...
[download] Got server HTTP error. Retrying fragment Seg1 (attempt 2 of 10)...
[download] Got server HTTP error. Retrying fragment Seg1 (attempt 3 of 10)...
[download] Got server HTTP error. Retrying fragment Seg1 (attempt 4 of 10)...
[download] Got server HTTP error. Retrying fragment Seg1 (attempt 5 of 10)...
[download] Got server HTTP error. Retrying fragment Seg1 (attempt 6 of 10)...
[download] Got server HTTP error. Retrying fragment Seg1 (attempt 7 of 10)...
[download] Got server HTTP error. Retrying fragment Seg1 (attempt 8 of 10)...
[download] Got server HTTP error. Retrying fragment Seg1 (attempt 9 of 10)...
[download] Got server HTTP error. Retrying fragment Seg1 (attempt 10 of 10)...
ERROR: giving up after 10 fragment retries
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "youtube-dl/youtube_dl/__main__.py", line 19, in <module>
    youtube_dl.main()
  File "youtube-dl/youtube_dl/__init__.py", line 433, in main
    _real_main(argv)
  File "youtube-dl/youtube_dl/__init__.py", line 423, in _real_main
    retcode = ydl.download(all_urls)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 1788, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 702, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 748, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 1432, in process_video_result
    self.process_info(new_info)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 1696, in process_info
    success = dl(filename, info_dict)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 1638, in dl
    return fd.download(name, info)
  File "youtube-dl/youtube_dl/downloader/common.py", line 354, in download
    return self.real_download(filename, info_dict)
  File "youtube-dl/youtube_dl/downloader/dash.py", line 74, in real_download
    append_url_to_file(segment_url, ctx['tmpfilename'], 'Seg%d' % i)
  File "youtube-dl/youtube_dl/downloader/dash.py", line 68, in append_url_to_file
    self.report_error('giving up after %s fragment retries' % fragment_retries)
  File "youtube-dl/youtube_dl/downloader/common.py", line 162, in report_error
    self.ydl.report_error(*args, **kargs)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 565, in report_error
    self.trouble(error_message, tb)
  File "youtube-dl/youtube_dl/YoutubeDL.py", line 527, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

[download] Got server HTTP error. Retrying fragment Seg2 (attempt 1 of 10)...
[download] Got server HTTP error. Retrying fragment Seg2 (attempt 2 of 10)...
[download] Got server HTTP error. Retrying fragment Seg2 (attempt 3 of 10)...
^C
ERROR: Interrupted by user

I'm not sure this behavior makes sense. I'm using --ignore-errors because it's very helpful when downloading entire channels, but this behavior 1) creates a flood of requests that makes me more likely to get banned 2) presumably doesn't get a complete video file anyway?

@yan12125 yan12125 added the bug label Aug 30, 2016
@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Aug 30, 2016

DashSegmentsFD should abort if the initialization segment is not available. Here's a WIP patch for this case:

diff --git a/youtube_dl/downloader/dash.py b/youtube_dl/downloader/dash.py
index 8bbab9d..c819cca 100644
--- a/youtube_dl/downloader/dash.py
+++ b/youtube_dl/downloader/dash.py
@@ -68,10 +68,15 @@ class DashSegmentsFD(FragmentFD):
                 self.report_error('giving up after %s fragment retries' % fragment_retries)
                 return False

+            return True
+
+        initialization_ok = True
         if initialization_url:
-            append_url_to_file(initialization_url, ctx['tmpfilename'], 'Init')
-        for i, segment_url in enumerate(segment_urls):
-            append_url_to_file(segment_url, ctx['tmpfilename'], 'Seg%d' % i)
+            initialization_ok = append_url_to_file(initialization_url, ctx['tmpfilename'], 'Init')
+
+        if initialization_ok:
+            for i, segment_url in enumerate(segment_urls):
+                append_url_to_file(segment_url, ctx['tmpfilename'], 'Seg%d' % i)

         self._finish_frag_download(ctx)

@dstftw Currently append_url_to_file in DashSegmentsFD returns False on failure and None on success. It's confusing. I suggest to change its return value to True and False only. Does it sound OK?

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Aug 30, 2016

DashSegmentsFD should abort if the initialization segment is not available.

should'nt it be generalized also to the case when the video doesn't have an intialization segment in MPD manifest at all.

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Aug 30, 2016

In commit a1a2257 I see you make initialization_url optional. Is there an example of valid DASH without initialization_url?

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Aug 30, 2016

i don't have an example, but making it optional was based on the standard.

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Aug 30, 2016

If the standard allows a DASH manifest without any initialization URL, it should be OK in youtube-dl, too.

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Aug 30, 2016

I don't think this default behavior should be changed. In most of the cases 404 is a temporary error for dash segments on YouTube and it's usually fixed by 1-2 subsequent retries.
#10451 makes this flow more explicit and accurate and introduces an additional option to abort on first unavailable fragment --abort-on-unavailable-fragment and thus allows to achieve what you want. Retry count per fragment can be configured with --fragment-retries.
ffmpeg also uses the same behavior for hls by default, i.e. skips unavailable fragments.

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Aug 30, 2016

I don't think this default behavior should be changed.

but it will be necessary for dash manifest with initialization to stop if it coudn't get initialization segment as it won't produce a working file.

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Aug 30, 2016

I guess it's format dependent whether the resulting file will be playable or not when the head segment is missing. Is there any statement in spec about this?

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Aug 30, 2016

I guess it's format dependent whether the resulting file will be playable or not when the head segment is missing.

it's possible, but the most used format in dash manifests is mp4 which didn't work without the initialization segment.

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Sep 1, 2016

Is there an example of valid DASH without initialization_url?

i found an example in the nytimes mpd manifests.

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Sep 3, 2016

i found an example in the nytimes mpd manifests.

Could you paste the URL?

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Sep 3, 2016

Here's my new proposal: #10542

@yan12125 yan12125 closed this in 919cf1a Sep 4, 2016
@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Sep 4, 2016

@ivan I've fixed it. In the next version the whole download process stops immediately if the first segment is unavailable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

4 participants
You can’t perform that action at this time.