max-filesize and match-filter not followed for certain streamed videos

[amishmm]

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.07.21. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2018.07.21

Before submitting an issue make sure you have:

• At least skimmed through the README, most notably the FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones
• Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

Verbose output as follows:

/usr/bin/youtube-dl -v --max-filesize 101m --match-filter '!is_live & !live & duration <= 960' -i --no-warnings --no-mark-watched 'https://www.youtube.com/watch?v=pbk3RR0uF0c'
[debug] System config: []
[debug] User config: ['--format', 'best[height<=480]/bestvideo[height<=480]+bestaudio/best[height<=720]/bestvideo[height<=720]+bestaudio/best', '--merge-output-format', 'mp4', '--restrict-filenames', '--no-mtime', '--console-title', '--write-sub', '--proxy', '127.0.0.1:8080', '--no-check-certificate', '--no-call-home']
[debug] Custom config: []
[debug] Command-line args: ['-v', '--max-filesize', '101m', '--match-filter', '!is_live & !live & duration <= 960', '-i', '--no-warnings', '--no-mark-watched', 'https://www.youtube.com/watch?v=pbk3RR0uF0c']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.07.21
[debug] Python version 3.6.6 (CPython) - Linux-4.17.8-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg n4.0.2, ffprobe n4.0.2, rtmpdump 2.4
[debug] Proxy map: {'http': '127.0.0.1:8080', 'https': '127.0.0.1:8080'}
[youtube] pbk3RR0uF0c: Downloading webpage
[youtube] pbk3RR0uF0c: Downloading video info webpage
[youtube] pbk3RR0uF0c: Downloading m3u8 information
[youtube] pbk3RR0uF0c: Downloading MPD manifest
[youtube] pbk3RR0uF0c: Downloading MPD manifest
[debug] Invoking downloader on 'https://manifest.googlevideo.com/...'
[download] Destination: safariLIVE_-_Sunset_Safari_-_July_23_2018-pbk3RR0uF0c.mp4
[debug] ffmpeg command line: ffmpeg -y -loglevel verbose -headers 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0 (Chrome)
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
' -i https://manifest.googlevideo.com/... -c copy -f mp4 -bsf:a aac_adtstoasc file:safariLIVE_-_Sunset_Safari_-_July_23_2018-pbk3RR0uF0c.mp4.part
ffmpeg version n4.0.2 Copyright (c) 2000-2018 the FFmpeg developers...

Description of your issue, suggested solution and other information

I am trying to download a youtube video (video ID = pbk3RR0uF0c) with match filter = '!is_live & !live & duration <= 960'.

That is download the video which is not live and is less than 16 minutes long. And also size should be < 101mb

The video however is more than 3 hours. And it was "streamed live" recently, just few minutes back.

As you can see in logs - youtube-dl goes ahead and downloads whole video (it actually downloaded 999MB - way more than 101MB limit that I supplied - then I manually killed process)

On further checking meta for the video pbk3RR0uF0c: (using --write-info-json --skip-download)
"duration": 1 ---- surprising!! (<=960 filter matches!)
"filesize": null --- again a surprise!! (should have failed here)

Thats probably the reason youtube-dl downloads the whole 3hr and 1GB (or more) file.

I believe for filesize: null cases - youtube-dl should stop downloading once limit of 101MB is crossed AND once 16mins of video is downloaded.

Thank you

PS: current workaround ... add !filesize to match-filter.

[amishmm]

The video above was streamed recently. May be youtube is still generating meta data for video and hence duration is 1 and filesize is null.

PS: just a random guess

[dstftw]

--max-filesize only works for plain HTTP downloads not HLS.

"duration": 1 ---- surprising!! (<=960 filter matches!)
"filesize": null --- again a surprise!! (should have failed here)

This is the metadata Youtube provides at the moment. Blame them.
As you already mentioned this is most likely due to video being processed after streaming or processing has just somewhat partially failed on their side.

[amishmm]

The thing is I am not actually downloading live stream but this video appears in my subscription list.

So when I download videos of my subscription - it starts downloading this stream as well, which is huge amount of data I do not want to download.

So as I said,
I believe for filesize: null cases - youtube-dl should stop downloading once limit of 101MB is crossed AND once 16mins of video is downloaded.

May be we should pass --max-filesize and duration to ffmpeg via "-fs" and "-t" options.

Or is there an easier way to exclude HLS streams? Say for example --no-hls