Machine fingerprinting issue? - ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests

[julcortez]

Checklist

• I'm asking a question
• I've looked through the README and FAQ for similar questions
• I've searched the bugtracker for similar questions including closed ones

Question

I have two headless machines in my internal network, one Ubuntu and another Raspbian. When using youtube-dl, the ubuntu one is giving me the error "ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests" while trying to download a specific link. The Raspbian machine works just great with that very specific link.

Resolving this error is important to me for two reasons: 1) I want to be able to be "as anonymous as possible" when downloading material through youtube-dl (and carrying out my normal browsing activity where I'm also subject to passive fingerprinting through google API and google analytics) and 2) solve the 429 error problem

I have carried out the following checks and actions in the Ubuntu machine and all these have failed to solve the problem and still gives the above 429 error:

1. I have confirmed that both share the same external IP. I have Windows machines in the same network where I can view yourube videos with no problem. No captchas needed.
2. I have updated youtube-dl to the newest version
3. I have used the --cookies flag to point to new cookies file (confirmed that the file gets populated with information)
4. I have used the --user-agent to provide the same string as the Raspbian machine
5. I have used the --rm-cache-dir flag
6. I have used the --source-address flag to change to different addresses
7. Have tried to change other flags.

Unfortunately no success. I know that this issue has been raised in the past, however no solution that solves the root of the problem has been provided. From my perspective, the root of the problem is to be able to have more control of the parameters sent in order to avoid this fingerprinting issue and of course solve the problem.

Below are the outputs using the Ubuntu (failed) and Raspbian (suceeded):

========= Ubuntu ==============

ubuntu@ubuntu:~$ youtube-dl --print-traffic -v https://www.youtube.com/watch?v=cFH5JgyZK1I [debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--print-traffic', '-v', 'https://www.youtube.com/watch?v=cFH5JgyZK1I']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.07.28
[debug] Python version 3.8.2 (CPython) - Linux-5.4.0-1015-raspi-aarch64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] cFH5JgyZK1I: Downloading webpage
send: b'GET /watch?v=cFH5JgyZK1I&gl=US&hl=en&has_verified=1&bpctr=9999999999&disable_polymer=true HTTP/1.1\r\nHost: www.youtube.com\r\nCookie: PREF=f1=50000000&f6=8&hl=en\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.18 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 429 Too Many Requests\r\n'
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: Cache-Control: no-cache
header: Content-Type: text/html; charset=utf-8
header: Strict-Transport-Security: max-age=31536000
header: Expires: Tue, 27 Apr 1971 19:44:06 GMT
header: X-Frame-Options: SAMEORIGIN
header: Content-Length: 57905
header: X-Content-Type-Options: nosniff
header: Date: Fri, 21 Aug 2020 02:15:31 GMT
header: Server: YouTube Frontend Proxy
header: X-XSS-Protection: 0
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Set-Cookie: VISITOR_INFO1_LIVE=IUELTgN8-h4; path=/; domain=.youtube.com; secure; expires=Wed, 17-Feb-2021 02:15:31 GMT; httponly; samesite=None
header: Set-Cookie: GPS=1; path=/; domain=.youtube.com; expires=Fri, 21-Aug-2020 02:45:31 GMT
header: Set-Cookie: YSC=7mENjWhn8T0; path=/; domain=.youtube.com; secure; httponly; samesite=None
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Set-Cookie: VISITOR_INFO1_LIVE=IUELTgN8-h4; path=/; domain=.youtube.com; secure; expires=Wed, 17-Feb-2021 02:15:31 GMT; httponly; samesite=None
header: Set-Cookie: GPS=1; path=/; domain=.youtube.com; expires=Fri, 21-Aug-2020 02:45:31 GMT
header: Set-Cookie: YSC=7mENjWhn8T0; path=/; domain=.youtube.com; secure; httponly; samesite=None
header: Alt-Svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
header: Connection: close
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by <HTTPError 429: 'Too Many Requests'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "./ytd/youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "./ytd/youtube-dl/youtube_dl/YoutubeDL.py", line 2238, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python3.8/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
response = self.parent.error(
File "/usr/lib/python3.8/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)

========= Raspbian ==============

pi@raspbian: $ youtube-dl --print-traffic -v https://www.youtube.com/watch?v=cFH5JgyZK1I

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--print-traffic', u'-v', u'https://www.youtube.com/watch?v=cFH5JgyZK1I']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.07.28
[debug] Python version 2.7.13 (CPython) - Linux-4.14.52-v7+-armv7l-with-debian-9.4
[debug] exe versions: ffmpeg 3.2.14-1, ffprobe 3.2.14-1, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] cFH5JgyZK1I: Downloading webpage
send: u'GET /watch?has_verified=1&bpctr=9999999999&hl=en&v=cFH5JgyZK1I&disable_polymer=true&gl=US HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3648.2 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,;q=0.7\r\nHost: www.youtube.com\r\nCookie: PREF=f1=50000000&f6=8&hl=en\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: no-cache
header: Strict-Transport-Security: max-age=31536000
header: X-Content-Type-Options: nosniff
header: Expires: Tue, 27 Apr 1971 19:44:06 GMT
header: X-Frame-Options: SAMEORIGIN
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: Content-Type: text/html; charset=utf-8
header: Content-Encoding: gzip
header: Date: Fri, 21 Aug 2020 02:17:39 GMT
header: Server: YouTube Frontend Proxy
header: X-XSS-Protection: 0
header: Set-Cookie: YSC=hSHzrpTYEeI; path=/; domain=.youtube.com; secure; httponly; samesite=None
header: Set-Cookie: GPS=1; path=/; domain=.youtube.com; expires=Fri, 21-Aug-2020 02:47:39 GMT
header: Set-Cookie: VISITOR_INFO1_LIVE=Z60yJZ5IzWY; path=/; domain=.youtube.com; secure; expires=Wed, 17-Feb-2021 02:17:39 GMT; httponly; samesite=None
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Alt-Svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
header: Connection: close
header: Transfer-Encoding: chunked
[youtube] {18} signature length 107, html5 player 530216c1
[youtube] cFH5JgyZK1I: Downloading player https://www.youtube.com/s/player/530216c1/player_ias.vflset/en_US/base.js
send: u'GET /s/player/530216c1/player_ias.vflset/en_US/base.js?disable_polymer=true HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3648.2 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,;q=0.7\r\nHost: www.youtube.com\r\nCookie: YSC=hSHzrpTYEeI; VISITOR_INFO1_LIVE=Z60yJZ5IzWY; PREF=f1=50000000&f6=8&hl=en; s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; GPS=1\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Accept-Ranges: bytes
header: Vary: Accept-Encoding, Origin
header: Content-Encoding: gzip
header: Content-Type: text/javascript
header: Content-Length: 461278
header: Date: Thu, 20 Aug 2020 15:20:16 GMT
header: Expires: Fri, 20 Aug 2021 15:20:16 GMT
header: Last-Modified: Thu, 20 Aug 2020 00:27:26 GMT
header: X-Content-Type-Options: nosniff
header: Server: sffe
header: X-XSS-Protection: 0
header: Cache-Control: public, max-age=31536000
header: Age: 39445
header: Alt-Svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
header: Connection: close

[youtube] {135} signature length 107, html5 player 530216c1
[youtube] {244} signature length 107, html5 player 530216c1
[youtube] {397} signature length 107, html5 player 530216c1
[youtube] {134} signature length 107, html5 player 530216c1
[youtube] {243} signature length 107, html5 player 530216c1
[youtube] {396} signature length 107, html5 player 530216c1
[youtube] {133} signature length 103, html5 player 530216c1
[youtube] cFH5JgyZK1I: Downloading player https://www.youtube.com/s/player/530216c1/player_ias.vflset/en_US/base.js
send: u'GET /s/player/530216c1/player_ias.vflset/en_US/base.js?disable_polymer=true HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3648.2 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,;q=0.7\r\nHost: www.youtube.com\r\nCookie: YSC=hSHzrpTYEeI; VISITOR_INFO1_LIVE=Z60yJZ5IzWY; PREF=f1=50000000&f6=8&hl=en; s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; GPS=1\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Accept-Ranges: bytes
header: Vary: Accept-Encoding, Origin
header: Content-Encoding: gzip
header: Content-Type: text/javascript
header: Content-Length: 461278
header: Date: Thu, 20 Aug 2020 15:24:37 GMT
header: Expires: Fri, 20 Aug 2021 15:24:37 GMT
header: Last-Modified: Thu, 20 Aug 2020 00:27:26 GMT
header: X-Content-Type-Options: nosniff
header: Server: sffe
header: X-XSS-Protection: 0
header: Cache-Control: public, max-age=31536000
header: Age: 39200
header: Alt-Svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
header: Connection: close
[youtube] {242} signature length 103, html5 player 530216c1
[youtube] {395} signature length 107, html5 player 530216c1
[youtube] {160} signature length 107, html5 player 530216c1
[youtube] {278} signature length 107, html5 player 530216c1
[youtube] {394} signature length 103, html5 player 530216c1
[youtube] {140} signature length 107, html5 player 530216c1
[youtube] {249} signature length 107, html5 player 530216c1
[youtube] {250} signature length 107, html5 player 530216c1
[youtube] {251} signature length 107, html5 player 530216c1
[debug] Default format spec: bestvideo+bestaudio/best
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[debug] Invoking downloader on u'https://r3---sn-upbvcv-a2cl.googlevideo.com/videoplayback?expire=1597997859&ei=wy4_X5H_Ms7JwgT5l6SoBg&ip=181.188.176.195&id=o-AKWdJgdQMTHOLul0iXAQ2V3fhdOjjCFfVm9XlmA0Dvk8&itag=135&aitags=133%2C134%2C135%2C160%2C242%2C243%2C244%2C278%2C394%2C395%2C396%2C397&source=youtube&requiressl=yes&mh=N-&mm=31%2C29&mn=sn-upbvcv-a2cl%2Csn-bvvbax-aphe&ms=au%2Crdu&mv=m&mvi=3&pl=24&nh=%2CEAE&initcwndbps=410000&vprv=1&mime=video%2Fmp4&gir=yes&clen=21875022&dur=266.840&lmt=1576402467978847&mt=1597976075&fvip=8&keepalive=yes&c=WEB&txp=5532432&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cnh%2Cinitcwndbps&lsig=AG3C_xAwRQIgU1sqqtrwqrgn3UFbYuqSIp9nbMcdkiG52qET8hg7zg0CIQDbHxXI7XJwF5BQj4jRtYM6dXVqmgi-soWCwy6i4UePQA%3D%3D&sig=AOq0QJ8wRQIgBnWxMMXhMMwh8gwCrX7_pooRKlddtTnEgwrGRiAoZo0CIQD4da7iEozLub5Iu3jnKLMhrAAB-yH3zsfxqJyOwXFRHA==&ratebypass=yes'
send: u'GET /videoplayback?expire=1597997859&ei=wy4_X5H_Ms7JwgT5l6SoBg&ip=181.188.176.195&id=o-AKWdJgdQMTHOLul0iXAQ2V3fhdOjjCFfVm9XlmA0Dvk8&itag=135&aitags=133%2C134%2C135%2C160%2C242%2C243%2C244%2C278%2C394%2C395%2C396%2C397&source=youtube&requiressl=yes&mh=N-&mm=31%2C29&mn=sn-upbvcv-a2cl%2Csn-bvvbax-aphe&ms=au%2Crdu&mv=m&mvi=3&pl=24&nh=%2CEAE&initcwndbps=410000&vprv=1&mime=video%2Fmp4&gir=yes&clen=21875022&dur=266.840&lmt=1576402467978847&mt=1597976075&fvip=8&keepalive=yes&c=WEB&txp=5532432&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cnh%2Cinitcwndbps&lsig=AG3C_xAwRQIgU1sqqtrwqrgn3UFbYuqSIp9nbMcdkiG52qET8hg7zg0CIQDbHxXI7XJwF5BQj4jRtYM6dXVqmgi-soWCwy6i4UePQA%3D%3D&sig=AOq0QJ8wRQIgBnWxMMXhMMwh8gwCrX7_pooRKlddtTnEgwrGRiAoZo0CIQD4da7iEozLub5Iu3jnKLMhrAAB-yH3zsfxqJyOwXFRHA==&ratebypass=yes HTTP/1.1\r\nAccept-Encoding: identity\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\nHost: r3---sn-upbvcv-a2cl.googlevideo.com\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3648.2 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,;q=0.7\r\nRange: bytes=0-9969775\r\n\r\n'
reply: 'HTTP/1.1 206 Partial Content\r\n'
header: Last-Modified: Sun, 15 Dec 2019 09:34:27 GMT
header: Content-Type: video/mp4
header: Date: Fri, 21 Aug 2020 02:18:14 GMT
header: Expires: Fri, 21 Aug 2020 02:18:14 GMT
header: Cache-Control: private, max-age=21265
header: Content-Range: bytes 0-9969775/21875022
header: Accept-Ranges: bytes
header: Content-Length: 9969776
header: Connection: keep-alive
header: Alt-Svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
header: Vary: Origin
header: X-Restrict-Formats-Hint: None
header: X-Content-Type-Options: nosniff
header: Server: gvs 1.0
[download] Destination: Talk Talk - It's My Life ( Official Video)-cFH5JgyZK1I.f135.mp4
[download] 1.0% of 20.86MiB at 70.79KiB/s ETA 04:58^C
ERROR: Interrupted by user

[ricardoandresrodriguezgamboa]

I have the same error, I don't find a stable solution, I use youtube-dl api deployed in heroku