TF1 missing video formats

[FM33]

Checklist

• [ x] I'm reporting a broken site support
• [ x] I've verified that I'm running youtube-dl version 2021.01.24.1
• [ x] I've checked that all provided URLs are alive and playable in a browser
• [ x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
• [x ] I've searched the bugtracker for similar issues including closed ones

Verbose log

[debug] System config: []
[debug] User config: ['-o', '/media/fabien/Archives/%(title)s-%(id)s.%(ext)s', '--hls-prefer-ffmpeg']
[debug] Custom config: []
[debug] Command-line args: ['-u', 'PRIVATE', '-p', 'PRIVATE', '-F', '--verbose', 'https://www.tf1.fr/tmc/quotidien-avec-yann-barthes/videos/quotidien-premiere-partie-du-26-janvier-2021-39755366.html']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.01.24.1
[debug] Python version 3.8.5 (CPython) - Linux-5.8.0-40-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Proxy map: {}
[TF1] quotidien-premiere-partie-du-26-janvier-2021-39755366: Downloading webpage
[wat.tv] 13762766: Downloading JSON metadata
[wat.tv] 13762766: Downloading JSON metadata
[wat.tv] 13762766: Downloading m3u8 information
[wat.tv] 13762766: Downloading MPD manifest
[info] Available formats for 13762766:
format code        extension  resolution note
**dash-audio=64000   m4a        audio only DASH audio   64k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-audio=128000  m4a        audio only DASH audio  128k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-video=200109  mp4        416x234    DASH video  200k , mp4_dash container, avc1.42C01E, 25fps, video only
hls-280            mp4        416x234     280k , avc1.42C01E@ 200k, 25.0fps, mp4a.40.2@ 64k (best)**

Description

Since last fix, yt-dl is not detecting all available formats, only low-quality ones. HD videos still exist because they are playable with web player.

[pingu1]

Note: the extensive list used to be something like

[info] Available formats for 13761436:
format code  extension  resolution note
hls-68       mp4        audio only   68k , mp4a.40.2@ 64k
hls-136      mp4        audio only  136k , mp4a.40.2@128k
hds-264-0    flv        416x234     264k
hds-264-1    flv        416x234     264k
hls-280-0    mp4        416x234     280k , avc1.42C01E@ 200k, 25.0fps, mp4a.40.2@ 64k
hls-280-1    mp4        416x234     280k , avc1.66.30@ 200k, 25.0fps, mp4a.40.2@ 64k
hds-528-0    flv        480x270     528k
hds-528-1    flv        480x270     528k
hls-560-0    mp4        480x270     560k , avc1.42C01E@ 400k, 25.0fps, mp4a.40.2@128k
hls-560-1    mp4        480x270     560k , avc1.66.30@ 400k, 25.0fps, mp4a.40.2@128k
hds-728-0    flv        640x360     728k
hds-728-1    flv        640x360     728k
hls-772-0    mp4        640x360     772k , avc1.42C01E@ 600k, 25.0fps, mp4a.40.2@128k
hls-772-1    mp4        640x360     772k , avc1.66.30@ 600k, 25.0fps, mp4a.40.2@128k
hds-1328-0   flv        1024x576   1328k
hds-1328-1   flv        1024x576   1328k
hls-1409-0   mp4        1024x576   1409k , avc1.4D401F@1200k, 25.0fps, mp4a.40.2@128k
hls-1409-1   mp4        1024x576   1409k , avc1.77.31@1200k, 25.0fps, mp4a.40.2@128k
hds-1828-0   flv        1024x576   1828k
hds-1828-1   flv        1024x576   1828k
hls-1939-0   mp4        1024x576   1939k , avc1.4D401F@1700k, 25.0fps, mp4a.40.2@128k
hls-1939-1   mp4        1024x576   1939k , avc1.77.31@1700k, 25.0fps, mp4a.40.2@128k
hds-2628-0   flv        1280x720   2628k
hds-2628-1   flv        1280x720   2628k
hls-2787-0   mp4        1280x720   2787k , avc1.4D401F@2500k, 25.0fps, mp4a.40.2@128k
hls-2787-1   mp4        1280x720   2787k , avc1.77.31@2500k, 25.0fps, mp4a.40.2@128k (best)

Some hints are available in issue #27201 : #27901 (comment)

[tampa-bay]

@pingu1, they did "fix" that method/moved that resource. At least on the videos I have tried since.

[pingu1]

Well, take this URL: https://www.tf1.fr/tmc/quotidien-avec-yann-barthes/videos/quotidien-premiere-partie-du-26-janvier-2021-39755366.html

Used as youtube-dl -F https://www.tf1.fr/tmc/quotidien-avec-yann-barthes/videos/quotidien-premiere-partie-du-26-janvier-2021-39755366.html

I tried to access the hls videos from the other subdomain you were mentioning in the other open issue, but I can only get 404s. I tried with the username/password, cookies, behind a FR proxy ... no luck, and I have little clue as where to find the HD formats.

Is there any chance to fetch all the format from the different resources?

[tampa-bay]

Sorry, I meant that tf1 "fixed" the method, moving the resources away from that location, which prevents users from finding it that way.

There likely is some digging that can be done to find it in an automated sense, but when I did the steps manually, it appears that TF1 has moved all their HD behind another json that requires login. When logging in, behind a proxy, in Network Inspection I can see a json at: https://mediainfo.tf1.fr/mediainfocombo/13762766?context=MYTF1&pver=XXXXX&platform=XXX&os=XXXX&osVersion=XXXXXX&topDomain=unknown which contains a link to the playlist for the HD versions.

It appears that they may want to drop "min_bitrate=XXXXXX" entirely and change max_bitrate to 2800000, in the mpd_url request.