Youtube occasionally serves unexpected signatures what leads to broken stream links (Error 403: Forbidden)

[kotvertolet]

Checklist

• I'm reporting a broken site support
• I've verified that I'm running youtube-dl version 2019.04.30
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-f', u'249', u'https://www.youtube.com/watch?v=JGwWNGJdvx8', u'-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.04.30
[debug] Python version 2.7.10 (CPython) - Darwin-18.2.0-x86_64-i386-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[youtube] JGwWNGJdvx8: Downloading webpage
[youtube] JGwWNGJdvx8: Downloading video info webpage
[youtube] {18} signature length 106, html5 player vflQTyJbT
[youtube] {313} signature length 110, html5 player vflQTyJbT
[youtube] {271} signature length 110, html5 player vflQTyJbT
[youtube] {137} signature length 106, html5 player vflQTyJbT
[youtube] {248} signature length 106, html5 player vflQTyJbT
[youtube] {136} signature length 110, html5 player vflQTyJbT
[youtube] {247} signature length 110, html5 player vflQTyJbT
[youtube] {398} signature length 110, html5 player vflQTyJbT
[youtube] {135} signature length 110, html5 player vflQTyJbT
[youtube] {244} signature length 110, html5 player vflQTyJbT
[youtube] {397} signature length 110, html5 player vflQTyJbT
[youtube] {134} signature length 110, html5 player vflQTyJbT
[youtube] {243} signature length 110, html5 player vflQTyJbT
[youtube] {396} signature length 110, html5 player vflQTyJbT
[youtube] {133} signature length 106, html5 player vflQTyJbT
[youtube] {242} signature length 106, html5 player vflQTyJbT
[youtube] {395} signature length 106, html5 player vflQTyJbT
[youtube] {160} signature length 110, html5 player vflQTyJbT
[youtube] {278} signature length 110, html5 player vflQTyJbT
[youtube] {394} signature length 110, html5 player vflQTyJbT
[youtube] {140} signature length 110, html5 player vflQTyJbT
[youtube] {171} signature length 110, html5 player vflQTyJbT
[youtube] {249} signature length 110, html5 player vflQTyJbT
[youtube] {250} signature length 106, html5 player vflQTyJbT
[youtube] {251} signature length 110, html5 player vflQTyJbT
[debug] Invoking downloader on u'https://r2---sn-3px8apox-3c2e.googlevideo.com/videoplayback?expire=1557436476&ei=3EPUXNOPJ4KU7ASL2rnIBw&ip=91.225.225.115&id=o-AEIiiIfvkEj8bYDf8S-qXlVSHYN5zWQTU6-7Ws8qVSO6&itag=249&source=youtube&requiressl=yes&mm=31%2C29&mn=sn-3px8apox-3c2e%2Csn-3c27sn7l&ms=au%2Crdu&mv=m&pl=22&nh=%2CIgpwcjAxLmticDAzKgkxMjcuMC4wLjE&initcwndbps=1108750&mime=audio%2Fwebm&gir=yes&clen=1609992&dur=263.241&lmt=1540766113290916&mt=1557414746&fvip=9&keepalive=yes&beids=23806823&c=WEB&txp=5511222&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mm%2Cmn%2Cms%2Cmv%2Cpl%2Cnh%2Cinitcwndbps&lsig=AHylml4wRQIhAIFF4sxU05JdOqOxN31e65peq4k8P2_4Iuwk5eDjb2yJAiAzYag_syL7fITSMptCp4DAyzh_Tik22z9Dk4mUicRJcg%3D%3D&signature=ALgxI2wwRgIhALiH4juW7ukPgRTeJjNC3VHMWEtWc5xSbxbiPP-Wuf-aAiEApk8sNDmp2Yi7Q0-75vbrWaLhlSrAqaYc4ozbzADnzqU=&ratebypass=yes'
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1915, in process_info
    success = dl(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1854, in dl
    return fd.download(name, info)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 364, in download
    return self.real_download(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/http.py", line 341, in real_download
    establish_connection()
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/http.py", line 109, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2227, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 403: Forbidden

Description

Hi guys! Yesterday I've found that youtube-dl from time to time serves me with broken links to the signatured audio/video streams, it happens not that often (2-3 times per 10-15 videos) but still quite uncomfortable so I decided to investigate this issue and here's what I've found - sometimes signatured video may have signature with format different than normal one and therefore (as I suppose) it can't be properly deciphered so the stream url leads to 403.

So instead of deciphered signature like this
549AFEF2E20684CF765993B37180D8F8C204EC09.63C2AE132DFDB8CD8A14DBD8310AE492969E4F2C
it may looks like this:
ALgxI2wwRQIgSD97zkwABo9JPAipp752LO891bTagLDq79fwYuq-7A4CIQD7lSb-j8BuvgmGEkCR2RV-gVtSIcz8NFz9JDIuJymoyw==
(ciphered signature was: 7LgxI2wwRQIgSD9AzkwABo9JPAipp752LO891bTagLDq79fwYuq-7A4CIQD7lSb-j8BuvgmGEkCRVRV-gutSIcz8NFz9JDI=Jymoyw==w=uw=2)

How to reproduce:

1. Run the following command:
   youtube-dl -f 249 'https://www.youtube.com/watch?v=JGwWNGJdvx8' -v(video id JGwWNGJdvx8 is not obligatory, feel free to use any other signatured video id)
2. If audio file was downloaded successfully delete the file
3. Try again
   Note: Usually it takes less than 10 attempts to reproduce this bug.

[sudovijay]

did notice this too, its weird, what i got as deciphered signature

ALgxI2wwRQIgQHWXbHQp-PO1OjO1Rrn1yYpXeovs8Pf0jNUk5TV3y3MCIQDZ1pNCw4H1tiNJ20br-ZRFLiRFFuPyprN7PxZvtu-Srg==

[charlie67]

Getting the same problem with the video you posted, my signature is ALgxI2wwRgIhAP0KE3wIc5sByjT_ZZZJerIiqupXkeTiDyk7X_YptivMAiEArPGvX-P3wZi1rlFkUkuORYicAAPI3n6qEU5m_-ShDTI=

[sudovijay]

there's a new pram too if you look closely -

lsig

https://redirector.googlevideo.com/videoplayback?expire=1557444664&ei=2GPUXO6HKNGE3LUP0JO38A8&ip=159.89.160.156&id=o-ALE4ObQe39olZt10oyF8vVbmaLSESN0l-Gx9h3KHrTt2&itag=171&source=youtube&requiressl=yes&mm=31,26&mn=sn-h5576nee,sn-cvh76nez&ms=au,onr&mv=m&pl=24&initcwndbps=111250&mime=audio%2Fwebm&gir=yes&clen=3937917&dur=263.205&lmt=1540766115306658&mt=1557422986&fvip=2&keepalive=yes&c=WEB&txp=5511222&sparams=expire,ei,ip,id,itag,source,requiressl,mime,gir,clen,dur,lmt&lsparams=mm,mn,ms,mv,pl,initcwndbps&lsig=AHylml4wRQIhAKLA-jGFYOK-BOAbGkOGJt6V-7m7QvGT5qCyDYODUGC4AiBvYu1UrqHfw_Umue-HeDdVfnrwZJbn1-wEktXb5RLvSg%3D%3D&signature=ALgxI2wwRQIgBiyfroos1dbUZu176HqUcDYlcKBINmVynC8mtiwj_woCIQCLKcC9F1tAOaPJ8gC5ukKWtxaAAME1Wd_vWWaejCc6iw==

[sudovijay]

it seems to be an old issue, old ones got closed though without much clarification;

#18841 #18927 #18937 #18950 #18804

[kotvertolet]

it seems to be an old issue, old ones got closed though without much clarification;

#18841 #18927 #18937 #18950 #18804

Looks like you're right - all the issues you've pointed except #18937 and #18804 has that weird signature like in my case.

I personally in my library (that is basically youtube-dl on java) added a straight forward workaround for this problem - I'm just checking if signature contains a dot character (valid signatures are always have it) and if not I'm starting all the magic again. I've tested if for dozens of times and discovered that 1 or 2 attempts are enough to receive a proper signature.