Twitter multi-video posts

[ethericist]

Twitter added a new feature which allows you to upload multiple videos to one post.

https://twitter.com/PrimeTobirama/status/1577759019724333058 an example post

When you attempt to download from this URL it will only download the first video instead of all 4.

[dirkf]

yt-dlp and Nitter (eg, https://nitter.eu/PrimeTobirama/status/1577759019724333058) also only find a single video.

The extractor uses an API call to get the videos, but neither the v1.1 API in use nor the newer v2 API has been updated to support returning 4 videos. Only a maximum of 4 photos or 1 video is supported.

If anyone can suggest how the extra videos could be found, we can try to implement it.

[altryne]

I'm getting this with twitter API v2

This is the full request:

curl --location --request GET 'https://api.twitter.com/2/tweets/1577700558638563328?expansions=attachments.media_keys&media.fields=' \

[altryne]

However in my case, youtube-dl goes into a loop instead of just returning the 1 :

[dirkf]

The loop can occur if there's no video or the first media item is a photo. Then the extractor looks for a media "card" structure. If that isn't found, it looks for the .entities.urls[0].expanded_url member of the JSON returned from the API: if that is just the original URL extraction goes into a loop.

[dirkf]

The media_key data in the API screenshot (please post text) might appear promising but we need another step to resolve the m_nnnnnnnnnnnnnnnnnnnn values. I suppose m is the media type code and the n...n is an ID that can be passed be used to construct a URL or API call.

[audas]

I just got a new error from Twitter on one video I have never seen before - maybe related to changes.

Downloading guest token [twitter] 1578103191874711552: Downloading JSON metadata [twitter] 1578090984604303362: Downloading JSON metadata ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by HTTPError());

It appears other videos are not affected by this.

Video URL
https://twitter.com/i/status/1578090984604303362

[dirkf]

...Unable to download JSON metadata: HTTP Error 404: Not Found...

Not repeatable with release or git master.

[dirkf]

Not as far as I know. No solution has been found in the corresponding yt-dlp thread, either.

[dirkf]

This is (essentially) what the extractor does, but it only gets a .extended_entities.media of length 1. Your JSON has length 4 as I originally hoped it would be.

The extractor gets a list of length 1 whether it uses api.twitter.com/1.1/statuses/show/1577759019724333058.json?tweet_mode=extended or api.twitter.com/1.1/statuses/show.json?id=1577759019724333058&tweet_mode=extended (as documented by Twitter).

There must be some other factor, headers or whatever, at work.

[dirkf]

Indeed:

$ python -m youtube_dl -v -F 'https://twitter.com/PrimeTobirama/status/1577759019724333058'
[debug] System config: [u'--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'-F', u'https://twitter.com/PrimeTobirama/status/1577759019724333058']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 2948a02f7
[debug] Python version 2.7.17 (CPython) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[twitter] 1577759019724333058: Downloading guest token
[twitter] 1577759019724333058: Downloading JSON metadata
[twitter] 1577759019724333058: Downloading m3u8 information
[twitter] 1577759019724333058: Downloading m3u8 information
[twitter] 1577759019724333058: Downloading m3u8 information
[twitter] 1577759019724333058: Downloading m3u8 information
[download] Downloading playlist: silver🎃 - 1577759019724333058
[twitter] playlist silver🎃 - 1577759019724333058: Collected 4 video ids (downloading 4 of them)
[download] Downloading video 1 of 4
[info] Available formats for 1577758955471790084:
format code  extension  resolution note
hls-256      mp4        280x270     256k , avc1.4d0015, mp4a.40.2
http-256     mp4        280x270     256k 
hls-832      mp4        374x360     832k , avc1.4d001e, mp4a.40.2
http-832     mp4        374x360     832k 
hls-2176     mp4        640x616    2176k , avc1.64001f, mp4a.40.2
http-2176    mp4        640x616    2176k  (best)
[download] Downloading video 2 of 4
[info] Available formats for 1577758955455102979:
format code  extension  resolution note
hls-256      mp4        480x270     256k , avc1.4d001e, mp4a.40.2
http-256     mp4        480x270     256k  (best)
[download] Downloading video 3 of 4
[info] Available formats for 1577758955442520067:
format code  extension  resolution note
hls-256      mp4        360x270     256k , avc1.4d001e, mp4a.40.2
http-256     mp4        360x270     256k 
hls-832      mp4        480x360     832k , avc1.4d001f, mp4a.40.2
http-832     mp4        480x360     832k  (best)
[download] Downloading video 4 of 4
[info] Available formats for 1577758955455094784:
format code  extension  resolution note
hls-256      mp4        358x270     256k , avc1.4d001e, mp4a.40.2
http-256     mp4        358x270     256k 
hls-832      mp4        476x360     832k , avc1.4d001f, mp4a.40.2
http-832     mp4        476x360     832k 
hls-2176     mp4        886x668    2176k , avc1.640020, mp4a.40.2
http-2176    mp4        886x668    2176k  (best)
[download] Finished downloading playlist: silver🎃 - 1577759019724333058
$

[klikevil]

Friend of mine also impacted by this, 2 videos attached to a tweet and only the first one downloads even when you directly tell it video 2

https://twitter.com/i/status/1790585249803440474

https://twitter.com/vxdb/status/1790585249803440474/video/2

yt-dlp and Nitter (eg, https://nitter.eu/PrimeTobirama/status/1577759019724333058) also only find a single video.

The extractor uses an API call to get the videos, but neither the v1.1 API in use nor the newer v2 API has been updated to support returning 4 videos. Only a maximum of 4 photos or 1 video is supported.

If anyone can suggest how the extra videos could be found, we can try to implement it.

Navigate to tweet ( https://twitter.com/vxdb/status/1790585249803440474 ), copy the video address -> navigate to that -( https://twitter.com/i/status/1790585249803440474 ) > click one of the videos then copy address from there ( https://twitter.com/vxdb/status/1790585249803440474/video/1 https://twitter.com/vxdb/status/1790585249803440474/video/2 ) -> download each playlist.

What's interesting is that yt-dlp when passed the -g argument sees both the videos it just only downloads the first one.

$ yt-dlp -g https://twitter.com/vxdb/status/1790585249803440474
https://video.twimg.com/ext_tw_video/1790585079137214464/pu/pl/avc1/720x1280/skErDveJ8RxnLqF4.m3u8?container=cmaf
https://video.twimg.com/ext_tw_video/1790585079137214464/pu/pl/mp4a/128000/EPPDvaULPKFW6AuS.m3u8?container=cmaf
https://video.twimg.com/ext_tw_video/1790585098036748288/pu/pl/avc1/720x1280/r2UZGJzR0A9NfkET.m3u8?container=cmaf
https://video.twimg.com/ext_tw_video/1790585098036748288/pu/pl/mp4a/128000/TgsqR6d9knITwNBE.m3u8?container=cmaf

To work around this you can use a shell function i wrote:

merge_m3u8 () 
{ 
    if [[ $# -lt 3 ]]; then
        echo -e "${FUNCNAME[0]} usage:\n${FUNCNAME[0]} <start sec> <duration> <output file> <url>" && return 1;
    else
        local start_sec="$1" duration="$2" url="$4" output_name="$3" add_urls=("${@:5}");
        if [[ ${#add_urls[@]} -ge 1 ]]; then
            for add in $(seq 0 1 $((${#add_urls[@]}-1)));
            do
                export other_urls="$(echo -n " -i ${add_urls[$add]} ")";
            done;
        fi;
#        debug statement.
#        declare -p start_sec duration output_name url add_urls other_urls 1>&2;
        ffmpeg -ss "${start_sec}" -i "${url}" ${other_urls} -t "${duration}" -c copy -bsf:a aac_adtstoasc -flags +global_header -y "${output_name}";
    fi
}

Example:

merge_m3u8 0 30 dbtest.mp4 'https://video.twimg.com/ext_tw_video/1790585098036748288/pu/pl/avc1/720x1280/r2UZGJzR0A9NfkET.m3u8?container=cmaf' 'https://video.twimg.com/ext_tw_video/1790585098036748288/pu/pl/mp4a/128000/TgsqR6d9knITwNBE.m3u8?container=cmaf'

This seemed to do the trick for them, fetches both video and audio hls and merges them at the same time 😊