Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[youtube] maxresdefault doesn't exist sometimes #340

Closed
pukkandan opened this issue May 23, 2021 · 8 comments
Closed

[youtube] maxresdefault doesn't exist sometimes #340

pukkandan opened this issue May 23, 2021 · 8 comments
Labels
bug Bug that is not site-specific site-enhancement Feature request for some website

Comments

@pukkandan
Copy link
Member

Originally posted by @Zirro here


I've received several 404's for maxresdefault URLs with this change, so unfortunately it doesn't seem to be true for every video. The best approach would probably be to initially request maxresdefault and look at the response code. If it's 404, try with hqdefault instead.

It could also be worth using a similar approach to check for the existence of a .webp-version of the thumbnail at the /vi_webp/-path before falling back to the .jpgˋ-version, though the difference in quality doesn't seem to be very significant here.


Here's one where there is no maxresdefault size at all: https://www.youtube.com/watch?v=pS4ca4GB-_c

https://i.ytimg.com/vi/pS4ca4GB-_c/maxresdefault.jpg (404)
https://i.ytimg.com/vi_webp/pS4ca4GB-_c/maxresdefault.webp (404)

Here's one where both JPEG and WebP-files are accessible at separate URLs: https://www.youtube.com/watch?v=goSEyGNfiPM

https://i.ytimg.com/vi/goSEyGNfiPM/maxresdefault.jpg (200)
https://i.ytimg.com/vi_webp/goSEyGNfiPM/maxresdefault.webp (200)

Here's one where only the JPEG-file is available: https://www.youtube.com/watch?v=xp-b4Ce4Mf4

https://i.ytimg.com/vi/xp-b4Ce4Mf4/maxresdefault.jpg (200)
https://i.ytimg.com/vi_webp/xp-b4Ce4Mf4/maxresdefault.webp (404)

I'm not sure if there are cases where only the WebP exists, though. Besides that it's mainly a question of whether checking for the .webp first should be done for the theoretical quality improvement, with the .jpg staying as fallback when there is no WebP available.

@pukkandan
Copy link
Member Author

pukkandan commented May 23, 2021

It's easy enough to check for 404s, but its also not really necessary. yt-dlp automatically downloads the next thumbnail if one isnt found

$ yt-dlp pS4ca4GB-_c --write-thumb --skip-down
[youtube] pS4ca4GB-_c: Downloading webpage
[info] pS4ca4GB-_c: Downloading 1 format(s): 248+251
[youtube] pS4ca4GB-_c: Downloading thumbnail  ...
WARNING: Unable to download thumbnail "https://i.ytimg.com/vi/pS4ca4GB-_c/maxresdefault.jpg": HTTP Error 404: Not Found
[youtube] pS4ca4GB-_c: Downloading thumbnail  ...
[youtube] pS4ca4GB-_c: Writing thumbnail to: Magnitude 9.1 Earthquake in Japan March 11, 2011 Compilation - Part 1 [pS4ca4GB-_c].jpg

but anyway, the second best thumbnail is actually https://i.ytimg.com/vi/pS4ca4GB-_c/sddefault.jpg and https://i.ytimg.com/vi_webp/pS4ca4GB-_c/sddefault.webp (which yt-dlp doesn't find), not hqdefault :/
I'll probably look at hard-coding more thumbnail URLs

@pukkandan
Copy link
Member Author

Just noting down the thumbnail URLs I know

https://i.ytimg.com/vi/pS4ca4GB-_c/maxresdefault.jpg
https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.jpg
https://i.ytimg.com/vi/pS4ca4GB-_c/sddefault.jpg
https://i.ytimg.com/vi/pS4ca4GB-_c/mqdefault.jpg
https://i.ytimg.com/vi/pS4ca4GB-_c/default.jpg
https://i.ytimg.com/vi/pS4ca4GB-_c/0.jpg
https://i.ytimg.com/vi/pS4ca4GB-_c/1.jpg
https://i.ytimg.com/vi/pS4ca4GB-_c/2.jpg
https://i.ytimg.com/vi/pS4ca4GB-_c/3.jpg

All of these also have webp versions like https://i.ytimg.com/vi_webp/<id>/<name>.webp

@Zirro
Copy link
Contributor

Zirro commented May 23, 2021

It's easy enough to check for 404s, but its also not really necessary. yt-dlp automatically downloads the next thumbnail if one isnt found

Ah, I was receiving the following error messages instead:

WARNING: Unable to download thumbnail "https://i.ytimg.com/vi/pS4ca4GB-_c/maxresdefault.jpg": HTTP Error 404: Not Found
[...]
WARNING: Skipping embedding the thumbnail because the file is missing.

It looks like this issue happens specifically when the --write-all-thumbnails flag is provided.

but anyway, the second best thumbnail is actually https://i.ytimg.com/vi/pS4ca4GB-_c/sddefault.jpg

Interesting, I don't remember having seen sddefault.jpg before. If you don't mind me asking, how/where did you find it?

@pukkandan
Copy link
Member Author

the thumbnail shown in embed is normally the highest resolution one. So I opened https://www.youtube.com/embed/pS4ca4GB-_c and checked the network requests

@pukkandan pukkandan added bug Bug that is not site-specific enhancement New feature or request labels May 23, 2021
@pukkandan pukkandan added site-enhancement Feature request for some website and removed enhancement New feature or request labels Jun 21, 2021
@ouovouo
Copy link

ouovouo commented Jun 22, 2021

live videos
https://i.ytimg.com/vi/<id>/maxresdefault_live.jpg

@pukkandan
Copy link
Member Author

The only way I've found to fully fix this is to hardcode the URLs. But I'm not sure I should fix it this way because of how many additional requests it takes to figure out which of the URLs are working

$ yt-dlp -v https://www.youtube.com/watch?v=pS4ca4GB-_c --list-thumb
[debug] Command-line config: ['-v', 'https://www.youtube.com/watch?v=pS4ca4GB-_c', '--list-thumb']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.07.07 (source)
[debug] Plugin Extractors: ['SamplePlugin']
[debug] Git HEAD: f0ff9979c6
[debug] Python version 3.9.6 (CPython 64bit) - Windows-10-10.0.19043-SP0
[debug] exe versions: ffmpeg 2021-04-11-git-309e3cc15c-full_build-www.gyan.dev, ffprobe 2021-04-11-git-309e3cc15c-full_build-www.gyan.dev, phantomjs 2.1.1
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=pS4ca4GB-_c
[youtube] pS4ca4GB-_c: Downloading webpage
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] [info] Testing thumbnail 21
[debug] [info] Unable to connect to thumbnail 21 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/sddefault.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 20
[debug] [info] Testing thumbnail 19
[debug] [info] Unable to connect to thumbnail 19 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/mqdefault.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 18
[debug] [info] Testing thumbnail 17
[debug] [info] Unable to connect to thumbnail 17 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/maxresdefault.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 16
[debug] [info] Unable to connect to thumbnail 16 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/maxresdefault.jpg" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 15
[debug] [info] Unable to connect to thumbnail 15 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 14
[debug] [info] Unable to connect to thumbnail 14 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/default.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 13
[debug] [info] Testing thumbnail 12
[debug] [info] Unable to connect to thumbnail 12 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/3.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 11
[debug] [info] Testing thumbnail 10
[debug] [info] Unable to connect to thumbnail 10 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/2.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 9
[debug] [info] Testing thumbnail 8
[debug] [info] Unable to connect to thumbnail 8 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/1.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 7
[debug] [info] Testing thumbnail 6
[debug] [info] Unable to connect to thumbnail 6 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/0.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 5
[info] Thumbnails for pS4ca4GB-_c:
ID width   height  URL
0  168     94      https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.jpg?sqp=-oaymwEbCKgBEF5IVfKriqkDDggBFQAAiEIYAXABwAEG&rs=AOn4CLCBQ4_ln39dFCbgt268t8lMJUtfdA
1  196     110     https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.jpg?sqp=-oaymwEbCMQBEG5IVfKriqkDDggBFQAAiEIYAXABwAEG&rs=AOn4CLACk7_pc2_UMDQ3R-p4Epk4q517Jw
2  246     138     https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.jpg?sqp=-oaymwEcCPYBEIoBSFXyq4qpAw4IARUAAIhCGAFwAcABBg==&rs=AOn4CLCoeij_0sT78RWLVeAwZ_oQZhb8RQ
3  336     188     https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.jpg?sqp=-oaymwEcCNACELwBSFXyq4qpAw4IARUAAIhCGAFwAcABBg==&rs=AOn4CLCC9H7tB1SUKxNjuEFpeLGHh4vZxQ
4  480     360     https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.jpg
5  unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/0.jpg
7  unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/1.jpg
9  unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/2.jpg
11 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/3.jpg
13 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/default.jpg
18 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/mqdefault.jpg
20 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/sddefault.jpg

PS: Dont mind the ordering, it can be fixed

The testing can be implemented lazily. So if only one thumbnail URL is requested, not all has to be tested.

$ yt-dlp -v https://www.youtube.com/watch?v=pS4ca4GB-_c --print thumbnail
[debug] Command-line config: ['-v', 'https://www.youtube.com/watch?v=pS4ca4GB-_c', '--print', 'thumbnail']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.07.07 (source)
[debug] Plugin Extractors: ['SamplePlugin']
[debug] Git HEAD: f0ff9979c6
[debug] Python version 3.9.6 (CPython 64bit) - Windows-10-10.0.19043-SP0
[debug] exe versions: ffmpeg 2021-04-11-git-309e3cc15c-full_build-www.gyan.dev, ffprobe 2021-04-11-git-309e3cc15c-full_build-www.gyan.dev, phantomjs 2.1.1
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=pS4ca4GB-_c
[youtube] pS4ca4GB-_c: Downloading webpage
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] [info] Testing thumbnail 21
[debug] [info] Unable to connect to thumbnail 21 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/sddefault.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 20
[debug] Default format spec: bestvideo*+bestaudio/best
[info] pS4ca4GB-_c: Downloading 1 format(s): 248+251
https://i.ytimg.com/vi/pS4ca4GB-_c/sddefault.jpg

But the issue is when writing infojson, all URLs must be resolved even if the user has no actual interest in the thumbnails

$ yt-dlp -v https://www.youtube.com/watch?v=pS4ca4GB-_c --write-info --skip-dow
[debug] Command-line config: ['-v', 'https://www.youtube.com/watch?v=pS4ca4GB-_c', '--write-info', '--skip-dow']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.07.07 (source)
[debug] Plugin Extractors: ['SamplePlugin']
[debug] Git HEAD: f0ff9979c6
[debug] Python version 3.9.6 (CPython 64bit) - Windows-10-10.0.19043-SP0
[debug] exe versions: ffmpeg 2021-04-11-git-309e3cc15c-full_build-www.gyan.dev, ffprobe 2021-04-11-git-309e3cc15c-full_build-www.gyan.dev, phantomjs 2.1.1
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=pS4ca4GB-_c
[youtube] pS4ca4GB-_c: Downloading webpage
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] [info] Testing thumbnail 21
[debug] [info] Unable to connect to thumbnail 21 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/sddefault.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 20
[debug] Default format spec: bestvideo*+bestaudio/best
[info] pS4ca4GB-_c: Downloading 1 format(s): 248+251
[info] Writing video metadata as JSON to: Magnitude 9.1 Earthquake in Japan March 11, 2011 Compilation - Part 1 [pS4ca4GB-_c].info.json
[debug] [info] Testing thumbnail 19
[debug] [info] Unable to connect to thumbnail 19 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/mqdefault.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 18
[debug] [info] Testing thumbnail 17
[debug] [info] Unable to connect to thumbnail 17 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/maxresdefault.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 16
[debug] [info] Unable to connect to thumbnail 16 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/maxresdefault.jpg" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 15
[debug] [info] Unable to connect to thumbnail 15 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 14
[debug] [info] Unable to connect to thumbnail 14 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/default.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 13
[debug] [info] Testing thumbnail 12
[debug] [info] Unable to connect to thumbnail 12 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/3.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 11
[debug] [info] Testing thumbnail 10
[debug] [info] Unable to connect to thumbnail 10 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/2.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 9
[debug] [info] Testing thumbnail 8
[debug] [info] Unable to connect to thumbnail 8 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/1.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 7
[debug] [info] Testing thumbnail 6
[debug] [info] Unable to connect to thumbnail 6 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/0.webp" - HTTP Error 404: Not Found. Skipping...
[debug] [info] Testing thumbnail 5

Another way to handle this to test the URLs only if the user requests so with --check-formats:

$ yt-dlp -v https://www.youtube.com/watch?v=pS4ca4GB-_c --list-thumb
[debug] Command-line config: ['-v', 'https://www.youtube.com/watch?v=pS4ca4GB-_c', '--list-thumb']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.07.07 (source)
[debug] Plugin Extractors: ['SamplePlugin']
[debug] Git HEAD: f0ff9979c6
[debug] Python version 3.9.6 (CPython 64bit) - Windows-10-10.0.19043-SP0
[debug] exe versions: ffmpeg 2021-04-11-git-309e3cc15c-full_build-www.gyan.dev, ffprobe 2021-04-11-git-309e3cc15c-full_build-www.gyan.dev, phantomjs 2.1.1
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=pS4ca4GB-_c
[youtube] pS4ca4GB-_c: Downloading webpage
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] Thumbnails for pS4ca4GB-_c:
ID width   height  URL
0  168     94      https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.jpg?sqp=-oaymwEbCKgBEF5IVfKriqkDDggBFQAAiEIYAXABwAEG&rs=AOn4CLCBQ4_ln39dFCbgt268t8lMJUtfdA
1  196     110     https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.jpg?sqp=-oaymwEbCMQBEG5IVfKriqkDDggBFQAAiEIYAXABwAEG&rs=AOn4CLACk7_pc2_UMDQ3R-p4Epk4q517Jw
2  246     138     https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.jpg?sqp=-oaymwEcCPYBEIoBSFXyq4qpAw4IARUAAIhCGAFwAcABBg==&rs=AOn4CLCoeij_0sT78RWLVeAwZ_oQZhb8RQ
3  336     188     https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.jpg?sqp=-oaymwEcCNACELwBSFXyq4qpAw4IARUAAIhCGAFwAcABBg==&rs=AOn4CLCC9H7tB1SUKxNjuEFpeLGHh4vZxQ
4  480     360     https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.jpg
5  unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/0.jpg
6  unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/0.webp
7  unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/1.jpg
8  unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/1.webp
9  unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/2.jpg
10 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/2.webp
11 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/3.jpg
12 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/3.webp
13 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/default.jpg
14 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/default.webp
15 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.webp
16 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/maxresdefault.jpg
17 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/maxresdefault.webp
18 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/mqdefault.jpg
19 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/mqdefault.webp
20 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/sddefault.jpg
21 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/sddefault.webp
$ yt-dlp -v https://www.youtube.com/watch?v=pS4ca4GB-_c --list-thumb --check-form
[debug] Command-line config: ['-v', 'https://www.youtube.com/watch?v=pS4ca4GB-_c', '--list-thumb', '--check-form']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.07.07 (source)
[debug] Plugin Extractors: ['SamplePlugin']
[debug] Git HEAD: f0ff9979c6
[debug] Python version 3.9.6 (CPython 64bit) - Windows-10-10.0.19043-SP0
[debug] exe versions: ffmpeg 2021-04-11-git-309e3cc15c-full_build-www.gyan.dev, ffprobe 2021-04-11-git-309e3cc15c-full_build-www.gyan.dev, phantomjs 2.1.1
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=pS4ca4GB-_c
[youtube] pS4ca4GB-_c: Downloading webpage
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] Testing thumbnail 21
[info] Unable to connect to thumbnail 21 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/sddefault.webp" - HTTP Error 404: Not Found. Skipping...
[info] Testing thumbnail 20
[info] Testing thumbnail 19
[info] Unable to connect to thumbnail 19 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/mqdefault.webp" - HTTP Error 404: Not Found. Skipping...
[info] Testing thumbnail 18
[info] Testing thumbnail 17
[info] Unable to connect to thumbnail 17 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/maxresdefault.webp" - HTTP Error 404: Not Found. Skipping...
[info] Testing thumbnail 16
[info] Unable to connect to thumbnail 16 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/maxresdefault.jpg" - HTTP Error 404: Not Found. Skipping...
[info] Testing thumbnail 15
[info] Unable to connect to thumbnail 15 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.webp" - HTTP Error 404: Not Found. Skipping...
[info] Testing thumbnail 14
[info] Unable to connect to thumbnail 14 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/default.webp" - HTTP Error 404: Not Found. Skipping...
[info] Testing thumbnail 13
[info] Testing thumbnail 12
[info] Unable to connect to thumbnail 12 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/3.webp" - HTTP Error 404: Not Found. Skipping...
[info] Testing thumbnail 11
[info] Testing thumbnail 10
[info] Unable to connect to thumbnail 10 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/2.webp" - HTTP Error 404: Not Found. Skipping...
[info] Testing thumbnail 9
[info] Testing thumbnail 8
[info] Unable to connect to thumbnail 8 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/1.webp" - HTTP Error 404: Not Found. Skipping...
[info] Testing thumbnail 7
[info] Testing thumbnail 6
[info] Unable to connect to thumbnail 6 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/0.webp" - HTTP Error 404: Not Found. Skipping...
[info] Testing thumbnail 5
[info] Testing thumbnail 4
[info] Testing thumbnail 3
[info] Testing thumbnail 2
[info] Testing thumbnail 1
[info] Testing thumbnail 0
[info] Thumbnails for pS4ca4GB-_c:
ID width   height  URL
0  168     94      https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.jpg?sqp=-oaymwEbCKgBEF5IVfKriqkDDggBFQAAiEIYAXABwAEG&rs=AOn4CLCBQ4_ln39dFCbgt268t8lMJUtfdA
1  196     110     https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.jpg?sqp=-oaymwEbCMQBEG5IVfKriqkDDggBFQAAiEIYAXABwAEG&rs=AOn4CLACk7_pc2_UMDQ3R-p4Epk4q517Jw
2  246     138     https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.jpg?sqp=-oaymwEcCPYBEIoBSFXyq4qpAw4IARUAAIhCGAFwAcABBg==&rs=AOn4CLCoeij_0sT78RWLVeAwZ_oQZhb8RQ
3  336     188     https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.jpg?sqp=-oaymwEcCNACELwBSFXyq4qpAw4IARUAAIhCGAFwAcABBg==&rs=AOn4CLCC9H7tB1SUKxNjuEFpeLGHh4vZxQ
4  480     360     https://i.ytimg.com/vi/pS4ca4GB-_c/hqdefault.jpg
5  unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/0.jpg
7  unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/1.jpg
9  unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/2.jpg
11 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/3.jpg
13 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/default.jpg
18 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/mqdefault.jpg
20 unknown unknown https://i.ytimg.com/vi/pS4ca4GB-_c/sddefault.jpg

This doesnt affect actual downloading of the thumbnail except for a warning:

$ yt-dlp -v https://www.youtube.com/watch?v=pS4ca4GB-_c --write-thumb --skip-down
[debug] Command-line config: ['-v', 'https://www.youtube.com/watch?v=pS4ca4GB-_c', '--write-thumb', '--skip-down']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.07.07 (source)
[debug] Plugin Extractors: ['SamplePlugin']
[debug] Git HEAD: f0ff9979c6
[debug] Python version 3.9.6 (CPython 64bit) - Windows-10-10.0.19043-SP0
[debug] exe versions: ffmpeg 2021-04-11-git-309e3cc15c-full_build-www.gyan.dev, ffprobe 2021-04-11-git-309e3cc15c-full_build-www.gyan.dev, phantomjs 2.1.1
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=pS4ca4GB-_c
[youtube] pS4ca4GB-_c: Downloading webpage
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] pS4ca4GB-_c: Downloading 1 format(s): 248+251
[youtube] pS4ca4GB-_c: Downloading thumbnail  ...
WARNING: Unable to download thumbnail "https://i.ytimg.com/vi/pS4ca4GB-_c/sddefault.webp": HTTP Error 404: Not Found
[youtube] pS4ca4GB-_c: Downloading thumbnail  ...
[youtube] pS4ca4GB-_c: Writing thumbnail to: Magnitude 9.1 Earthquake in Japan March 11, 2011 Compilation - Part 1 [pS4ca4GB-_c].jpg

The issue then becomes that the URLs in --print thumbnail, --list-thumbnail and even the infojson may be unreliable unless used with --check-formats (same as #402)

$ yt-dlp -v https://www.youtube.com/watch?v=pS4ca4GB-_c --print thumbnail
[debug] Command-line config: ['-v', 'https://www.youtube.com/watch?v=pS4ca4GB-_c', '--print', 'thumbnail']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.07.07 (source)
[debug] Plugin Extractors: ['SamplePlugin']
[debug] Git HEAD: f0ff9979c6
[debug] Python version 3.9.6 (CPython 64bit) - Windows-10-10.0.19043-SP0
[debug] exe versions: ffmpeg 2021-04-11-git-309e3cc15c-full_build-www.gyan.dev, ffprobe 2021-04-11-git-309e3cc15c-full_build-www.gyan.dev, phantomjs 2.1.1
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=pS4ca4GB-_c
[youtube] pS4ca4GB-_c: Downloading webpage
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] pS4ca4GB-_c: Downloading 1 format(s): 248+251
https://i.ytimg.com/vi/pS4ca4GB-_c/sddefault.webp
$ yt-dlp -v https://www.youtube.com/watch?v=pS4ca4GB-_c --print thumbnail --check-form
[debug] Command-line config: ['-v', 'https://www.youtube.com/watch?v=pS4ca4GB-_c', '--print', 'thumbnail', '--check-form']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.07.07 (source)
[debug] Plugin Extractors: ['SamplePlugin']
[debug] Git HEAD: f0ff9979c6
[debug] Python version 3.9.6 (CPython 64bit) - Windows-10-10.0.19043-SP0
[debug] exe versions: ffmpeg 2021-04-11-git-309e3cc15c-full_build-www.gyan.dev, ffprobe 2021-04-11-git-309e3cc15c-full_build-www.gyan.dev, phantomjs 2.1.1
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=pS4ca4GB-_c
[youtube] pS4ca4GB-_c: Downloading webpage
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] Testing thumbnail 21
[info] Unable to connect to thumbnail 21 URL "https://i.ytimg.com/vi/pS4ca4GB-_c/sddefault.webp" - HTTP Error 404: Not Found. Skipping...
[info] Testing thumbnail 20
[debug] Default format spec: bestvideo*+bestaudio/best
[info] Testing format 248
[download] Destination: D:\Programs\Source\yt-dlp\tmpd092ax9p.tmp
[download] 100% of 10.00KiB in 00:00
[info] Testing format 251
[download] Destination: D:\Programs\Source\yt-dlp\tmpnnmdfod4.tmp
[download] 100% of 10.00KiB in 00:00
[info] pS4ca4GB-_c: Downloading 1 format(s): 248+251
https://i.ytimg.com/vi/pS4ca4GB-_c/sddefault.jpg

@Zirro
Copy link
Contributor

Zirro commented Jul 14, 2021

I suspect that most users are interested only in finding the highest quality thumbnail, with the smaller sizes being less relevant. Perhaps it'd be okay to only include details on the best available thumbnail in the infojson by default, omitting the others completely unless additional flags are provided?

It's probably also fine to treat failed thumbnail requests as expected behaviour, by not printing errors/warnings about them (unless in verbose mode).

pukkandan added a commit that referenced this issue Jul 19, 2021
* Also fix bug where `_test_url` was being ignored

Ref: https://stackoverflow.com/a/20542029
Related: #340
pukkandan added a commit that referenced this issue Oct 18, 2021
Partially reverts cca80fe and 0ba692a

Unless `--check-formats` is specified, this causes yt-dlp to return incorrect thumbnail urls.
See #340 (comment), #402

But the overhead in general use is not worth it

Closes #694, #725
@pukkandan
Copy link
Member Author

I don't know why I didn't think of this before. I can just put the best "known" thumbnail as thumbnail while still populating thumbnails with all possible thumbnails.

Then --get-thumbnail will give you a correct (but not the best) thumbnail without needing to check all the URLs while --write-thumbnail (due to it's in-build fallback logic) always gives the best quality thumbnail.

If anyone wants to get the best thumbnail URL and don't mind the slowdown, they can use --print %(thumbnails.-1.url)s --check-all-formats

Lesmiscore added a commit to ytdl-patched/ytdl-patched that referenced this issue Oct 25, 2021
* 'master' of https://github.com/yt-dlp/yt-dlp:
  [instagram] Fix bug in ab2ffab (#1403)
  Approximate filesize from bitrate Closes #1400
  [youtube] Populate `thumbnail` with the best "known" thumbnail Closes #402, Related: yt-dlp/yt-dlp#340 (comment)
  Separate `--check-all-formats` from `--check-formats`
nixxo pushed a commit to nixxo/yt-dlp that referenced this issue Nov 22, 2021
* The thumbnail URLs are hard-coded and their actual existence is tested lazily
* Added option `--no-check-formats` to not test them

Closes yt-dlp#340, Related: yt-dlp#402, yt-dlp#337, ytdl-org/youtube-dl#29049
nixxo pushed a commit to nixxo/yt-dlp that referenced this issue Nov 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Bug that is not site-specific site-enhancement Feature request for some website
Projects
None yet
Development

No branches or pull requests

3 participants