Can't download twitter videos from private accounts

[pcislocked]

Checklist

• I've read bugs section in FAQ

• I'm reporting a broken site support issue

• I've verified that I'm running youtube-dl version 2021.01.03

• I've checked that all provided URLs are alive and playable in a browser

• I've checked that all URLs and arguments with special characters are properly quoted or escaped

• I've searched the bugtracker for similar bug reports including closed ones

Verbose log

link:https://twitter.com/i/status/1131278212288458757
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--console-title', '--cookies', 'cookies.txt', '--verbose', '--no-playlist', '--merge-output-format', 'mp4', '-f', 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best', '-o', 'C:\\Users\\pcislocked\\Desktop\\%(id)s.%(ext)s', '--', 'https://twitter.com/i/status/1131278212288458757']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2021.01.03
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg 4.3.1, ffprobe 4.3.1
[debug] Proxy map: {}
[twitter] 1131278212288458757: Downloading guest token
ERROR: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmp7n_m4vnl\build\youtube_dl\extractor\common.py", line 632, in _request_webpage
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmp7n_m4vnl\build\youtube_dl\YoutubeDL.py", line 2248, in urlopen
  File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
  File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
  File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
  File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
  File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default

Press any key to continue . . .

Description

youtube-dl is unable to download videos from private twitter accounts. I can't provide a twitter account, but you can easily reproduce the same issue by trying to download any video from any private twitter account you follow. I put all twitter.com cookies in cookies.txt(which includes so-called the "guest token"), but that did not help.

[fossdd]

The line where it crashes is here:

youtube-dl/youtube_dl/extractor/common.py

Line 632 in 19ec468

return self._downloader.urlopen(url_or_request)

[JordanWeatherby]

Want to add that this isn't a problem directly related to downloading from a private account.

Any attempt to pass --cookies alongside a twitter url (even public accounts) causes this error.

I don't really have enough of a grasp of the youtube-dl implementation of cookies to understand why this is happening exactly but thought I'd mention what I'd discovered for anyone else to have a look at

[mischif]

I'm going to look more into this over the weekend, but I can tell you the first 403 is because the tool is including cookies in the request to https://api.twitter.com/1.1/guest/activate.json; removing them makes the request for a guest token succeed, but the next request fails due to lack of a csrf token (that could be me though, will add more cookies to my jar and retry).

EDIT: the root cause is the missing CSRF token - if you include the auth_token and ct0 cookies, and copy the ct0 cookie into an x-csrf-token header, you can download videos from private accounts.

EDIT 2: As a workaround, you could call the tool like so:
youtube-dl --cookies cookies.txt --add-header x-csrf-token:ct0goeshere https://twitter.com/xxx/status/yyy