[Twitter] Extract mp4 urls via mobile User-Agent

[dban0001]

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.04.11. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2017.04.11

Before submitting an issue make sure you have:

• At least skimmed through README and most notably FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

I want to have mp4 files from the --dump-json output. When I paste a twitter link, only a m3u8 url returns – although the video is available as mp4.
Example:
Twitter URL: https://twitter.com/news_al3alm/status/852138619213144067
Download URL (mp4, not from youtube-dl): https://video.twimg.com/amplify_video/852077943283097602/vid/640x360/e23l2ZGkx7rHBOY2.mp4

What youtube-dl does:

youtube-dl "https://twitter.com/freethenipple/status/643211948184596480" -f mp4 --dump-json --verbose | jq .url
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://twitter.com/freethenipple/status/643211948184596480', u'-f', u'mp4', u'--dump-json', u'--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out None, pref UTF-8
[debug] youtube-dl version 2017.04.11
[debug] Python version 2.7.12 - Linux-4.4.0-21-generic-x86_64-with-LinuxMint-18-sarah
[debug] exe versions: ffmpeg N-82999-g72d6101, ffprobe N-82999-g72d6101, rtmpdump 2.4
[debug] Proxy map: {}
"https://video.twimg.com/ext_tw_video/643211870443208704/pu/pl/480x480/i6PK7NIOtQ-_egbq.m3u8"

It also doesn't work if I provide -f best or -f mp4.

[yan12125]

Twitter uses M3U8 on desktop browsers. Extra codes are necessary for MP4 links. See also #11569 and #10668.

[singhpratyush]

Any example of User-Agent and Tweet URL combination that has mp4 link in it?

[yan12125]

@singhpratyush: Not sure what you want. Hasn't @dban0001 already mentioned one?

Twitter URL: https://twitter.com/news_al3alm/status/852138619213144067
Download URL (mp4, not from youtube-dl): https://video.twimg.com/amplify_video/852077943283097602/vid/640x360/e23l2ZGkx7rHBOY2.mp4

[singhpratyush]

@yan12125: I was looking for a User Agent for which mp4 link is present in the response body.

$ http <some_url> User-Agent:<some_user_agent>
...
<video ... >
    ...
    <source src="https://video.twimg.com/amplify_video/<ID>/vid/<resolution>/<some_stirng>.mp4" ...>
</video>
...

I have seen that these links are generated in https://mobile.twitter.com/... type pages on a normal browser but they are rendered using React.

[yan12125]

mp4 URLs are not embedded in web pages. Here's the flow:

1. Fetch the script with "bearer" authorization token. In the web page there's a line:

<script crossorigin="anonymous" onerror="showFailureMessage('https://ma-0.twimg.com/twitter-assets/responsive-web/web/ltr/main.5b6bf12947d7a3a6.js');" src="https://ma-0.twimg.com/twitter-assets/responsive-web/web/ltr/main.5b6bf12947d7a3a6.js"></script>

The "bearer" token can be found in https://ma-0.twimg.com/twitter-assets/responsive-web/web/ltr/main.5b6bf12947d7a3a6.js:

BEARER_TOKEN: "AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA",

2. Get the guest token from the web page:

<script>
  document.cookie = decodeURIComponent("gt=869422393856360448; Max-Age=10800; Domain=.twitter.com; Path=/");
</script>

869422393856360448 is the desired token.
3. Issue a request to Twitter's API server and parse the resultant JSON:

$ curl 'https://api.twitter.com/2/timeline/conversation/852138619213144067.json' -H 'Authorization: Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA' -H 'x-guest-token: 869422393856360448' --compressed | jq '.globalObjects.tweets["852138619213144067"].extended_entities.media[0].video_info.variants'

The result is:

[
  {
    "content_type": "application/x-mpegURL",
    "url": "https://video.twimg.com/amplify_video/852077943283097602/pl/W1KWzvz9IQtK55D2.m3u8"
  },
  {
    "bitrate": 832000,
    "content_type": "video/mp4",
    "url": "https://video.twimg.com/amplify_video/852077943283097602/vid/640x360/e23l2ZGkx7rHBOY2.mp4"
  },
  {
    "bitrate": 320000,
    "content_type": "video/mp4",
    "url": "https://video.twimg.com/amplify_video/852077943283097602/vid/320x180/wSdL-TycEjbLuseN.mp4"
  }
]

Here 852138619213144067 is the tweet ID, which is cut from the URL.

It turns out that a mobile User-Agent string is not necessary. It just helps on analyzing the behavior of twitter video pages.

[isklikas]

Hey, any more details on getting the guest token? I do not seem to be able to get this from this page, it just doesn't appear on there. I've only found gt, when opening mobile.twitter.com, in an anonymous window and inspecting session cookies, but I totally can't find a way to make curl get it. Because even if it's logged out, even with no cookies, no cache policy, this cookie will not be sent, even though it will be available on a browser.

[yan12125]

@isklikas: Check the work-in-progress version at https://github.com/yan12125/youtube-dl/tree/wip-twitter-12726, especially the guest_token variable in twitter.py.
Note that it works for https://twitter.com/news_al3alm/status/852138619213144067 but I haven't checked it doesn't break other videos.

[isklikas]

The way you suggested above, along with getting the guest token from a modern mobile client (that's the only part I've missed myself), actually seems to work for every video for me! I wrote it in Objective C, where I set the same headers and ephemeral session, which is just the name for "anonymous window" in iOS NSURLSessions. Thank you very much!

…

On 15 Jun 2017, at 14:48, Yen Chi Hsuan ***@***.***> wrote: There's a work-in-progress version at https://github.com/yan12125/youtube-dl/tree/wip-twitter-12726. It works for https://twitter.com/news_al3alm/status/852138619213144067 but I haven't checked it doesn't break other videos. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[yan12125]

This will be fixed in the next version. Sorry for being late as I was quite busy last month.

[Kikobeats]

I moved mi original comment to #17233

[BlackComte]

@yan12125 hey brother

mp4 URLs are not embedded in web pages. Here's the flow:

1. Fetch the script with "bearer" authorization token. In the web page there's a line:

<script crossorigin="anonymous" onerror="showFailureMessage('https://ma-0.twimg.com/twitter-assets/responsive-web/web/ltr/main.5b6bf12947d7a3a6.js');" src="https://ma-0.twimg.com/twitter-assets/responsive-web/web/ltr/main.5b6bf12947d7a3a6.js"></script>

"In the web page there's a line", I haven't found this line in web page, what is the web page url? Just like: https://twitter.com/news_al3alm/status/852138619213144067?

[yan12125]

Most likely Twitter mobile pages have changed over time. Also, I consider this approach stale as HLS streams are served even on mobile devices now.