Extraction fails on YouTube if --user-agent=Android option is passed

[Flashwalker]

Checklist

• [*] I'm reporting a broken site support issue
• [*] I've verified that I'm running youtube-dl version 2021.12.17
• [*] I've checked that all provided URLs are alive and playable in a browser
• [*] I've checked that all URLs and arguments with special characters are properly quoted or escaped
• [*] I've searched the bugtracker for similar bug reports including closed ones
• [*] I've read bugs section in FAQ

Verbose log

 $ wget https://yt-dl.org/downloads/latest/youtube-dl -O ~/bin/youtube-dl
 $ chmod a+rx ~/bin/youtube-dl
 $ hash -r

ok:

 $ ~/bin/youtube-dl --verbose -F 'https://www.youtube.com/watch?v=MejbOFk7H6c'
 
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'-F', u'https://www.youtube.com/watch?v=MejbOFk7H6c']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 2.7.18 (CPython) - Linux-5.4.44-x86_64-with-Pop-20.04-focal
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] MejbOFk7H6c: Downloading webpage
[youtube] MejbOFk7H6c: Downloading player c8b8a173
[info] Available formats for MejbOFk7H6c:
format code  extension  resolution note
249          webm       audio only tiny   46k , webm_dash container, opus @ 46k (48000Hz), 1.28MiB
250          webm       audio only tiny   59k , webm_dash container, opus @ 59k (48000Hz), 1.65MiB
251          webm       audio only tiny  114k , webm_dash container, opus @114k (48000Hz), 3.18MiB
140          m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2@129k (44100Hz), 3.61MiB
394          mp4        256x144    144p   74k , mp4_dash container, av01.0.00M.08@  74k, 24fps, video only, 2.08MiB
278          webm       256x144    144p   90k , webm_dash container, vp9@  90k, 24fps, video only, 2.53MiB
160          mp4        256x144    144p   92k , mp4_dash container, avc1.4d400c@  92k, 24fps, video only, 2.58MiB
395          mp4        426x240    240p  166k , mp4_dash container, av01.0.00M.08@ 166k, 24fps, video only, 4.64MiB
242          webm       426x240    240p  193k , webm_dash container, vp9@ 193k, 24fps, video only, 5.38MiB
133          mp4        426x240    240p  228k , mp4_dash container, avc1.4d4015@ 228k, 24fps, video only, 6.35MiB
396          mp4        640x360    360p  307k , mp4_dash container, av01.0.01M.08@ 307k, 24fps, video only, 8.57MiB
243          webm       640x360    360p  346k , webm_dash container, vp9@ 346k, 24fps, video only, 9.64MiB
134          mp4        640x360    360p  450k , mp4_dash container, avc1.4d401e@ 450k, 24fps, video only, 12.53MiB
397          mp4        854x480    480p  553k , mp4_dash container, av01.0.04M.08@ 553k, 24fps, video only, 15.41MiB
244          webm       854x480    480p  564k , webm_dash container, vp9@ 564k, 24fps, video only, 15.70MiB
135          mp4        854x480    480p  671k , mp4_dash container, avc1.4d401e@ 671k, 24fps, video only, 18.68MiB
247          webm       1280x720   720p 1048k , webm_dash container, vp9@1048k, 24fps, video only, 29.19MiB
136          mp4        1280x720   720p 1106k , mp4_dash container, avc1.4d401f@1106k, 24fps, video only, 30.81MiB
398          mp4        1280x720   720p 1139k , mp4_dash container, av01.0.05M.08@1139k, 24fps, video only, 31.71MiB
18           mp4        640x360    360p  702k , avc1.42001E, 24fps, mp4a.40.2 (44100Hz), 19.55MiB
22           mp4        1280x720   720p 1235k , avc1.64001F, 24fps, mp4a.40.2 (44100Hz) (best)

not ok:

 $ ~/bin/youtube-dl --verbose --user-agent=Android -F 'https://www.youtube.com/watch?v=MejbOFk7H6c'
 
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'--user-agent=Android', u'-F', u'https://www.youtube.com/watch?v=MejbOFk7H6c']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 2.7.18 (CPython) - Linux-5.4.44-x86_64-with-Pop-20.04-focal
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] MejbOFk7H6c: Downloading webpage
[youtube] MejbOFk7H6c: Downloading API JSON
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/home/user/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/home/user/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/home/user/bin/youtube-dl/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/home/user/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 1733, in _real_extract
    self._sort_formats(formats)
  File "/home/user/bin/youtube-dl/youtube_dl/extractor/common.py", line 1374, in _sort_formats
    raise ExtractorError('No video formats found')
ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

Extraction fails on YouTube if --user-agent=Android option is passed

[Flashwalker]

OK so, don't do that?

okay-mem.jpg

[Flashwalker]

It worked all these years, so... I don't really remember why I used it in my bash scripts.
I'm using yt-dlp now, works fine, despite player extraction response warnings

[dirkf]

Here are some example real Android UAs.

Eg:

$ python3 -m youtube_dl -v -F 'https://www.youtube.com/watch?v=MejbOFk7H6c' --user-agent 'Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.133 Mobile Safari/535.19'
[debug] System config: ['--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '-F', 'https://www.youtube.com/watch?v=MejbOFk7H6c', '--user-agent', 'Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.133 Mobile Safari/535.19']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 1ff18694c
[debug] Python version 3.5.2 (CPython) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[youtube] MejbOFk7H6c: Downloading webpage
[debug] [youtube] Decrypted nsig n4I0vUobDbLBO8x4 => 7OBnI7g-mOQ1dA
[debug] [youtube] Decrypted nsig V_s1ueY4SJOSStTg => 2fapIcTbnfzEwQ
[youtube] MejbOFk7H6c: Downloading API JSON
[info] Available formats for MejbOFk7H6c:
format code  extension  resolution note
249          webm       audio only tiny   46k , webm_dash container, opus @ 46k (48000Hz), 1.28MiB
250          webm       audio only tiny   59k , webm_dash container, opus @ 59k (48000Hz), 1.65MiB
251          webm       audio only tiny  114k , webm_dash container, opus @114k (48000Hz), 3.18MiB
140          m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2@129k (44100Hz), 3.61MiB
394          mp4        256x144    144p   74k , mp4_dash container, av01.0.00M.08@  74k, 24fps, video only, 2.08MiB
278          webm       256x144    144p   90k , webm_dash container, vp9@  90k, 24fps, video only, 2.53MiB
160          mp4        256x144    144p   92k , mp4_dash container, avc1.4d400c@  92k, 24fps, video only, 2.58MiB
395          mp4        426x240    240p  166k , mp4_dash container, av01.0.00M.08@ 166k, 24fps, video only, 4.64MiB
242          webm       426x240    240p  193k , webm_dash container, vp9@ 193k, 24fps, video only, 5.38MiB
133          mp4        426x240    240p  228k , mp4_dash container, avc1.4d4015@ 228k, 24fps, video only, 6.35MiB
396          mp4        640x360    360p  307k , mp4_dash container, av01.0.01M.08@ 307k, 24fps, video only, 8.57MiB
243          webm       640x360    360p  346k , webm_dash container, vp9@ 346k, 24fps, video only, 9.64MiB
134          mp4        640x360    360p  450k , mp4_dash container, avc1.4d401e@ 450k, 24fps, video only, 12.53MiB
397          mp4        854x480    480p  553k , mp4_dash container, av01.0.04M.08@ 553k, 24fps, video only, 15.41MiB
244          webm       854x480    480p  564k , webm_dash container, vp9@ 564k, 24fps, video only, 15.70MiB
135          mp4        854x480    480p  671k , mp4_dash container, avc1.4d401e@ 671k, 24fps, video only, 18.68MiB
247          webm       1280x720   720p 1048k , webm_dash container, vp9@1048k, 24fps, video only, 29.19MiB
136          mp4        1280x720   720p 1106k , mp4_dash container, avc1.4d401f@1106k, 24fps, video only, 30.81MiB
398          mp4        1280x720   720p 1139k , mp4_dash container, av01.0.05M.08@1139k, 24fps, video only, 31.71MiB
18           mp4        640x360    360p  702k , avc1.42001E, 24fps, mp4a.40.2 (44100Hz), 19.55MiB
22           mp4        1280x720   720p 1235k , avc1.64001F, 24fps, mp4a.40.2 (44100Hz) (best)
$

At a minimum, if for some reason you want to mention Android in the UA string, you need to use something like Mozilla/5.0 (Android 4) (a version number is needed). Or just Mozilla/5.0 works. Perhaps your Android UA was supposed to make YT serve formats for pages that need an age-check, but I don't think that works now, if it ever did.

[pukkandan]

imo, any issue caused by (mis)use of --add-header, --user-agent etc should just be closed, even if the UA itself is valid. These are advanced options meant to be used by those who know what they are doing.