YoutubeDL DASH manifest is downloaded BEFORE basic checks

[ad33329]

• I've verified and I assure that I'm running youtube-dl 2017.08.13
• At least skimmed through the README, most notably the FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones
• Bug report (encountered problems with youtube-dl)

$ youtube-dl -a watchlist.txt --dateafter 'today-1day' --download-archive download-history.txt --cache-dir cache -v                        [18:26:21]
[debug] System config: ['--no-mtime']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-a', 'watchlist.txt', '--dateafter', 'today-1day', '--download-archive', 'download-history.txt', '--cache-dir', 'cache', '-v']
[debug] Batch file urls: ['https://www.youtube.com/channel/UCxseO_JzIiiJENauW2RmcJQ', 'https://www.youtube.com/channel/UCe_vXdMrHHseZ_esYUskSBw', 'https://www.youtube.com/channel/UCZYTClx2T1of7BRZ86-8fow', 'https://www.youtube.com/watch?v=5CHpBG0BYlU', 'https://www.youtube.com/channel/UCxt9Pvye-9x_AIcb1UtmF1Q', 'https://www.youtube.com/channel/UCFKDEp9si4RmHFWJW1vYsMA', 'https://www.youtube.com/channel/UCEOXxzW2vU0P-0THehuIIeg', 'https://www.youtube.com/channel/UCZJUuPIFk7VsgboB4cFwsSA', 'https://www.youtube.com/playlist?list=PLaDrN74SfdT6Z-jpVvGNEqaMdXHTLDdnI', 'https://www.youtube.com/playlist?list=PLaDrN74SfdT6duuVl_8qxJ5eaaPHRX_ij']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.08.13
[debug] Python version 3.6.2 - Linux-4.12.0-1-hardened-x86_64-with-arch-Arch-Linux
[debug] exe versions: ffmpeg 3.3.2, ffprobe 3.3.2, rtmpdump 2.4
[debug] Proxy map: {}
[youtube:channel] UCxseO_JzIiiJENauW2RmcJQ: Downloading channel page
[youtube:playlist] UUxseO_JzIiiJENauW2RmcJQ: Downloading webpage
[download] Downloading playlist: Uploads from barnabydixon
[youtube:playlist] UUxseO_JzIiiJENauW2RmcJQ: Downloading page #1
[youtube:playlist] playlist Uploads from barnabydixon: Downloading 108 videos
[download] Downloading video 1 of 108
[youtube] LI_34wuZPlc: Downloading webpage
[youtube] LI_34wuZPlc: Downloading video info webpage
[youtube] LI_34wuZPlc: Extracting video information
[youtube] LI_34wuZPlc: Downloading MPD manifest
[debug] Default format spec: bestvideo+bestaudio/best
[download] 2017-07-15 upload date is not in range 2017-08-12 - 9999-12-31
[download] Downloading video 2 of 108
[youtube] CN_m7LFglF4: Downloading webpage
[youtube] CN_m7LFglF4: Downloading video info webpage
[youtube] CN_m7LFglF4: Extracting video information
[youtube] CN_m7LFglF4: Downloading MPD manifest
[debug] Default format spec: bestvideo+bestaudio/best
[download] 2017-07-08 upload date is not in range 2017-08-12 - 9999-12-31
[download] Downloading video 3 of 108
[youtube] pha140nl-aA: Downloading webpage
[youtube] pha140nl-aA: Downloading video info webpage
[youtube] pha140nl-aA: Extracting video information
[youtube] pha140nl-aA: Downloading MPD manifest
[debug] Default format spec: bestvideo+bestaudio/best
[download] 2017-07-01 upload date is not in range 2017-08-12 - 9999-12-31
[download] Downloading video 4 of 108
[youtube] 6CTYdYeLLAc: Downloading webpage
[youtube] 6CTYdYeLLAc: Downloading video info webpage
[youtube] 6CTYdYeLLAc: Extracting video information
[youtube] 6CTYdYeLLAc: Downloading MPD manifest
[debug] Default format spec: bestvideo+bestaudio/best
[download] 2017-06-24 upload date is not in range 2017-08-12 - 9999-12-31
[download] Downloading video 5 of 108
[youtube] 3QzUtnbbvvo: Downloading webpage
^C
ERROR: Interrupted by user
FAIL: 1

(I terminated the program with Ctrl+C, as it running to completion was unhelpful for debugging.)

I've noticed that youtube-dl will proceed to download the MPD manifest for every video before it checks if the upload date is in range. This looks like unintended behavior to me.

I'm not exactly sure what the MPD manifest is, but from my cursory research, it seems to be just metadata used to allow the client to dynamically request a specific format from the server, depending on network connectivity. (Correct me if I'm wrong on this, please!)

I've also tried running the same command as above, except with --youtube-skip-dash-manifest, and it's still able to tell which videos it's supposed to not be downloading just fine. So the MPD manifest seems not to contain any information needed determine whether youtube-dl should even attempt to download the video in the first place.

[ad33329]

@dstftw
I did a search of the bug tracker and wasn't able to find the duplicate. Please reference it?

[ad33329]

@dstftw
I did a more thorough search, and found this. I'm guessing that's what you were talking about?

I really do think this should be classified as a bug rather than feature request. If a program is much slower than it should be due to optimization oversights (which also waste bandwidth with unnecessary requests), that seems like a bug to my way of thinking. Also a rather important one. I'm surprised you didn't prioritize this.

But anyway, since y'all have put this on the backburner for 2 years, I guess I'll have to fix it myself! 😛

[Hrxn]

I've also tried running the same command as above, except with --youtube-skip-dash-manifest, and it's still able to tell which videos it's supposed to not be downloading just fine. So the MPD manifest seems not to contain any information needed determine whether youtube-dl should even attempt to download the video in the first place.

I think the question is if this is still true for older/newer/different videos.
It might be correct for most videos, but it might not be for all.

[ad33329]

@Hrxn

So, you're saying YouTube's metadata is stored inconsistently, and sometimes the dash manifest contains stuff like, say, the upload date?
Edit: is there any actual documentation on the manifest, or have the project maintainers just been reverse engineering it on the fly? I can find surprisingly little on the formatting of it on DuckDuckGo.

[Hrxn]

I honestly don't know, but on the other hand, yes, I wouldn't be surprised about inconsistency at all when it comes to YouTube.

[ad33329]

Aww. Guess I'll ask the peeps on IRC and see if they know anything. I would like to eventually try to make a pull request if I can to resolve the wasted bandwidth issue. Thanks for your input, @Hrxn.