Feature request: use ffprobe/avprobe to fetch extra metadata for use in _sort_formats

[sebma]

Hi, I have noticed that the "-f bestaudio" does not really download the best audio quality :

$ youtube-dl --ignore-config -o "%(title)s__%(format_id)s__%(id)s.%(ext)s" -f bestaudio --restrict-filenames https://www.youtube.com/watch?v=NLYggpNyQuo
[youtube] NLYggpNyQuo: Downloading webpage
[youtube] NLYggpNyQuo: Extracting video information
[youtube] NLYggpNyQuo: Downloading DASH manifest
[download] Kacou_Severin_-Le_secret_de_l_onction_1__140__NLYggpNyQuo.m4a has already been downloaded
[download] 100% of 136.33MiB
[ffmpeg] Correcting container in "Kacou_Severin_-Le_secret_de_l_onction_1__140__NLYggpNyQuo.m4a"

Using the options "-f best -x" gets a file with better audio quality :

$ youtube-dl --ignore-config -o "%(title)s__%(format_id)s__%(id)s.%(ext)s" -f best -x -k --restrict-filenames https://www.youtube.com/watch?v=NLYggpNyQuo
[youtube] NLYggpNyQuo: Downloading webpage
[youtube] NLYggpNyQuo: Extracting video information
[youtube] NLYggpNyQuo: Downloading DASH manifest
[download] Kacou_Severin_-Le_secret_de_l_onction_1__22__NLYggpNyQuo.mp4 has already been downloaded
[download] 100% of 375.25MiB
[avconv] Destination: Kacou_Severin_-Le_secret_de_l_onction_1__22__NLYggpNyQuo.m4a

$ avprobe Kacou_Severin_-Le_secret_de_l_onction_1__140__NLYggpNyQuo.m4a 2>&1 | grep Audio:
Stream #0.0(und): Audio: aac, 44100 Hz, stereo, fltp, 125 kb/s (default)

$ avprobe Kacou_Severin_-Le_secret_de_l_onction_1__22__NLYggpNyQuo.mp4 2>&1 | grep Audio:
Stream #0.1(und): Audio: aac, 44100 Hz, stereo, fltp, 191 kb/s (default)

[yan12125]

Current behavior is correct. -f bestaudio picks the one with the best quality from audio-only files YouTube serves. The file from -f best -x is locally processed, which is not in the range "audio-only files from the server".

It's a good idea to extract higher quality audio from video+audio files. However, there's technical obstacles. AFAIK it's impossible to determine the audio quality in videos before actual downloading, for YouTube does not provide such information. If you know how, feel free to propose the method.

[sebma]

Hi,

Actually there are ways to determine the audio quality in videos before actual downloading:
you can either use avprobe/ffprobe or mediainfo.

In JSON format:

avprobe -v 0 -show_format -show_streams -of json $(youtube-dl -g $url)

In INI format:

avprobe -v 0 -show_format -show_streams -of ini $(youtube-dl -g $url)

In XML format:

mediainfo -f --Output=XML $(youtube-dl -g $url)

Can you please consider reopening this issue ?

Sebastien.

[yan12125]

I've tested your command on my FreeBSD virtual machine. The command is:

touch log.txt && \
    netstat -I em0 -b >> log.txt && \
    python youtube-dl --get-url --format best NLYggpNyQuo >> log.txt && \
    netstat -I em0 -b >> log.txt && \
    ffprobe -v 0 -show_format -show_streams $(python youtube-dl --get-url --format best NLYggpNyQuo) >> log.txt && \
    netstat -I em0 -b >> log.txt

And the result can be found at https://gist.github.com/yan12125/550087c250f373ccdb25
The first youtube-dl command consumes 51775326 - 51690745 = 84581 bytes, and the second command, youtube-dl with ffprobe, consumes 55122078 - 51775326 = 3346752 bytes. That is, ffprobe actually downloads part of the video for checking formats

[sebma]

How about mediainfo ?

[yan12125]

Roughly the same - 3868948 bytes in mediainfo. See https://gist.github.com/yan12125/11d66ba0a73d3d844385 for the detailed log. This time I use an Arch Linux VM because I can't make mediainfo work on FreeBSD.

[sebma]

Is it such a big deal downloading 3.2MiB (for ffprobe) instead of downloading the whole file ?

[sebma]

BTW : What youtube url did you use for your test with ffprobe ?

[yan12125]

https://www.youtube.com/watch?v=NLYggpNyQuo - the video in your first post.

I think it is. The command takes quite a few seconds for me. On slower networks, youtube-dl would look frozen.

[sebma]

As youtube-dl is dedicated to downloading video files which by definition are big, I don't think testing on slow networks is a normal use case.
And compared to the full size of the file which is 375MiB, ffprobe downloads only 1% to get the information, isn't it good ?

[yan12125]

it's indeed good in your case, but may not be always true for everyone. People may use youtube-dl just to download bestaudio, or just subtitles. 3MB is not a small amout in these cases.

With another much smaller video https://www.youtube.com/watch?v=cbjMwKLE-RE, ffprobe still downloads 3335011 bytes. It's 31% of the whole video (best). I guess the amount to download is hardcoded somewhere in the program. As a result, the detection approach is not suitable for all videos.

[sebma]

I understand.

Why not add a special option just for my case :)
No just kidding.

But how about adding an option like "--probe-bestaudio" which, if passed on the command line, would call ffprobe to find the audio bitrate ?

[yan12125]

OK a new option may be fine.

[yan12125]

A side note: this feature depends on #7034.