Large Twitter Galleries Not Fully Downloading #2226

ghost · 2022-01-25T17:15:35Z

I tried to download every MP4 from this gallery (NSFW), and it only went as far back as this tweet (also NSFW). After that tweet, it just stopped and acted as though it had downloaded the entire gallery, meaning that any older tweets, such as this one, were excluded.

If I don't use the link for the media tab, it stops at an even more recent tweet (NSFW).

For reference, the command I ran was gallery-dl "https://twitter.com/furui_1111/media" --filter "extension in ('mp4')"

The text was updated successfully, but these errors were encountered:

Zoodee · 2022-01-25T17:32:08Z

This is a limit on twitter's end, unfortunately.

#1396

ImportTaste · 2022-01-28T17:20:03Z

I personally ended up having to make a PowerShell script that would cycle through two week intervals since the user's registration date using Twitter search queries, with one day of overlap in both from: and until: so nothing gets skipped over (with include:nativeretweets in the search and &f=live instead of &f=top in the url).

Even still, I don't think that will get everything, because Twitter is dumb like that. It caps out retrieving a user's timeline at around 3200 tweets (and that includes retweets).

tux93 · 2022-02-10T20:29:15Z

What usually works for me is using a search query to download:
gallery-dl "https://twitter.com/search?q=(from:USER)"

wankio · 2022-05-25T14:33:51Z

twitter shadow hide all nsfw from search feature so i think there's no way to download all, you must use
https://stevesie.com/apps/twitter-api/scrape/tweets/by-user

Twi-Hard · 2022-05-25T15:00:31Z

From my experience, it seems people usually don't flag stuff as "sensitive content". I've seen that with both art and irl stuff. I've been using the api to get the total tweet count of 100% nsfw accounts and compared it against how much I could get from the search results and it's usually most of the tweets. I tried searching a nsfw tag in the browser and tweets marked by Twitter as "sensitive content" (ones that require you to verify you want to see it) still popped up occasionally.
Here's how they determine what's allowed in search results: https://help.twitter.com/en/using-twitter/twitter-search-not-working

wankio · 2022-05-25T16:56:07Z

it's been blocked since 2020

biggestsonicfan · 2022-08-09T09:54:47Z

Bumping this because I'm trying to consolidate my various twitter archives. I have a lot of content that was downloaded from twMediaDownloader and I planned to merge this content with gallery-dl using twitter-click-and-save to minimize file duplication by hardlinking across the drive.

My example case is casulcasulcasul with the following settings:

"twitter":
        {
            "sleep": 3.0,
            "sleep-request": 3.0,
            "archive": "/run/media/xxx/bfd18/dl/gallery-dl/sql/twitter.sqlite3",
            "archive-format": "{author[name]}—{date:%Y.%m.%d}—{tweet_id}—{filename}",
            "username": "xxx",
            "password": "yyy",
            "cards": false,
            "conversations": false,
            "quoted": false,
            "replies": true,
            "retweets": false,
            "text-tweets": false,
            "twitpic": false,
            "users": "timeline",
            "videos": true,
            "filename": "[twitter] {author[name]}—{date:%Y.%m.%d}—{tweet_id}—{filename}.{extension}"
        },

As this account seems to fall under 1000 media tweets, I try gallery-dl https://twitter.com/casulcasulcasul/media.

Out of the 970 media files twMediaDownloader calculated (using dryrun), gallery-dl using the above command downloaded 944, seemingly omitting anything earlier than November 2019.

Using gallery-dl "https://twitter.com/search?q=(from:casulcasulcasul)" we get a bit further (but the process is much slower):
This results in 968 files...

I don't quite know what/where the issue is, but the two tweets gallery-dl seems to miss are this one and this one. Manually grabbing those tweets with gallery-dl downloads just fine.

EDIT: I guess theoretically, you could use twMediaDownloader to generate a list of media tweets and use the .csv file it provides as input for gallery-dl. 🤔

EDIT2: Some issues are stating to include filter:media in your gallery-dl command but I have never once had this work. However, I have found f=media works, so the full command would be gallery-dl "https://twitter.com/search?q=(from:username)&f=media" and the quotes are important for single command line but aren't needed if an input file is used. This command gets more media than just gallery-dl https://twitter.com/username/media and is faster than gallery-dl "https://twitter.com/search?q=(from:username)" alone.

nisehime · 2022-08-11T13:02:43Z

Are you sure gallery-dl "https://twitter.com/search?q=(from:username)&f=media" makes a difference? From what I see it shouldn't.

You can use both https://twitter.com/casulcasulcasul/media and https://twitter.com/search?q=(from:casulcasulcasul) links. Once you download first one, copy the ID of the last downloaded tweet and put it in a search like this: https://twitter.com/search?q=from:casulcasulcasul+max_id:ID_HERE. To speed up the process you can add filter:links
(https://twitter.com/search?q=from:casulcasulcasul+max_id:ID_HERE+filter:links).

Or you can download latest artifact and simply paste https://twitter.com/casulcasulcasul if you don't want to bother with 2 links yourself.

biggestsonicfan · 2022-08-14T12:14:18Z

I will scream it from the rooftops:

xxx@DESKTOP-KLLQALU:~> gallery-dl https://twitter.com/search?q=from:casulcasulcasul+max_id:1081020936185274371+filter:links
twitter: NotFoundError: Requested user could not be found
xxx@DESKTOP-KLLQALU:~> gallery-dl https://twitter.com/search?q=from:casulcasulcasul+max_id:1081020936185274371
twitter: NotFoundError: Requested user could not be found
xxx@DESKTOP-KLLQALU:~> gallery-dl https://twitter.com/search?q=(from:casulcasulcasul)+max_id:1081020936185274371+filter:links
bash: syntax error near unexpected token `('
xxx@DESKTOP-KLLQALU:~> gallery-dl https://twitter.com/search?q=(from:casulcasulcasul)+max_id:1081020936185274371+filter:links
bash: syntax error near unexpected token `('
xxx@DESKTOP-KLLQALU:~> gallery-dl https://twitter.com/search?q=(from:casulcasulcasul)+max_id:1081020936185274371
bash: syntax error near unexpected token `('
xxx@DESKTOP-KLLQALU:~> gallery-dl https://twitter.com/search?q=(from:casulcasulcasul)
bash: syntax error near unexpected token `('
xxx@DESKTOP-KLLQALU:~> gallery-dl https://twitter.com/search?q=(from:casulcasulcasul)+max_id:1081020936185274371+filter:links"
bash: syntax error near unexpected token `('
xxx@DESKTOP-KLLQALU:~> gallery-dl "https://twitter.com/search?q=(from:casulcasulcasul)+max_id:1081020936185274371+filter:links"
twitter: NotFoundError: Requested user could not be found
xxx@DESKTOP-KLLQALU:~> gallery-dl "https://twitter.com/search?q=(from:casulcasulcasul)+max_id:1081020936185274371"
twitter: NotFoundError: Requested user could not be found
xxx@DESKTOP-KLLQALU:~> gallery-dl "https://twitter.com/search?q=(from:casulcasulcasul)+filter:links"
twitter: NotFoundError: Requested user could not be found
xxx@DESKTOP-KLLQALU:~> gallery-dl "https://twitter.com/search?q=(from:casulcasulcasul)"
/run/media/xxx/bfd18/dl/gallery-dl/twitter/casulcasulcasul/[twitter] casulcasulcasul—2022.08.13—1558577888386920448—FaEnVbxUIAAuR8J.jpg
/run/media/xxx/bfd18/dl/gallery-dl/twitter/casulcasulcasul/[twitter] casulcasulcasul—2022.08.12—1558220785201713152—FZ-ytOZakAEydm7.jpg

Addding + or filter: will not work with my install of gallery-dl. --version output: 1.23.0-dev (linux)

mikf · 2022-08-14T12:38:06Z

@cglmrfreeman use %20 or plain spaces instead of + signs

$ gallery-dl https://twitter.com/search?q=from:casulcasulcasul%20max_id:1081020936185274371%20filter:links
/tmp/twitter/casulcasulcasul/1081020936185274371_1.jpg
/tmp/twitter/casulcasulcasul/1071670086094643201_1.jpg
...
$ gallery-dl "https://twitter.com/search?q=from:casulcasulcasul max_id:1081020936185274371 filter:links"
/tmp/twitter/casulcasulcasul/1081020936185274371_1.jpg
/tmp/twitter/casulcasulcasul/1071670086094643201_1.jpg
...

biggestsonicfan · 2022-08-14T12:50:54Z

Huh, that one worked. I don't think I've ever seen anyone suggest that before. It's always "copy the twitter search url" https://twitter.com/search?q=from%3Acasulcasulcasul+max_id%3A1081020936185274371+filter%3Alinks which does not work or use the + or & signs that seemingly throw Requested user could not be found.

I will def be using this from now on, thanks!

nisehime · 2022-08-14T15:05:48Z

You probably should have put the link in double quotes "https://twitter.com/search?q=from:casulcasulcasul+max_id:ID_HERE+filter:links" to get + working.

But as mikf said plain spaces are fine too. In double quotes as well.

mikf · 2022-08-14T15:17:35Z

No, + signs as space replacements do not work in gallery-dl.

The function that parses query parameters does not "support" them, meaning it just returns + as is and does not replace them with a space character as might be expected.

nisehime · 2022-08-14T15:33:20Z

I see. Well, it still works with twitter specifically. Pluses in a query string are just ignored by twitter (or treated as spaces).

mikf · 2022-08-14T15:58:48Z

Oh, so the "NotFoundError"s are a bug introduced with 77bdd8f.

This commit splits search queries by whitespace only, and throws an error because there is no user named casulcasulcasul+max_id:1081020936185274371+filter:links

biggestsonicfan · 2022-08-14T16:26:45Z

Ah I only recently started using gallery-dl for twitter archiving and I definitely updated after that, so that might explain it.

nisehime · 2022-08-14T16:31:22Z

I see x2, I'm on latest stable ver, so I didn't notice. I thought you would leave the behavior for search as it was. I guess you should also consider that there can be multiple from: in a query if you haven't already. Also @ can be used instead of from:

... and do not raise exception if searched user does not exist

biggestsonicfan · 2022-09-03T04:52:24Z

For smaller galleries gallery-dl "https://twitter.com/search?q=from:Cotonus filter:links" does not grab nearly as much as gallery-dl https://twitter.com/Cotonus, and gallery-dl "https://twitter.com/search?q=from:Cotonus filter:media" only grabs 3 files. Twitter filters really suck these days.

nisehime · 2022-09-03T07:29:22Z

If you mean retweets you should add include:nativeretweets in the search

biggestsonicfan · 2022-09-03T17:30:08Z

I don't mean retweets.
gallery-dl "https://twitter.com/search?q=from:Cotonus filter:links" - 28 files
gallery-dl https://twitter.com/Cotonus - 32 files
gallery-dl "https://twitter.com/search?q=from:Cotonus filter:media" - 3 files

nisehime · 2022-09-03T19:57:09Z

Yeah, there's 2 posts which don't appear in the search at all. Even without filters.

biggestsonicfan · 2023-01-24T05:21:40Z

Popping back in here to say after fairly extensive testing, gallery-dl https://twitter.com/username is actually giving the maximum number of results at this point.

wankio · 2023-01-24T12:30:23Z

Popping back in here to say after fairly extensive testing, gallery-dl https://twitter.com/username is actually giving the maximum number of results at this point.

usually username and username/media, but pretty sure if their twitter have so many retweet and media, you can't get all, tries some 5-10k tweet to see, that's twitter limit

mikf added a commit that referenced this issue Aug 18, 2022

[twitter] unescape '+' in search queries (#2226)

264f133

... and do not raise exception if searched user does not exist

mikf closed this as completed Dec 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Large Twitter Galleries Not Fully Downloading #2226

Large Twitter Galleries Not Fully Downloading #2226

ghost commented Jan 25, 2022 •

edited by ghost

Zoodee commented Jan 25, 2022

ImportTaste commented Jan 28, 2022

tux93 commented Feb 10, 2022

wankio commented May 25, 2022

Twi-Hard commented May 25, 2022

wankio commented May 25, 2022

biggestsonicfan commented Aug 9, 2022 •

edited

nisehime commented Aug 11, 2022

biggestsonicfan commented Aug 14, 2022 •

edited

mikf commented Aug 14, 2022

biggestsonicfan commented Aug 14, 2022

nisehime commented Aug 14, 2022 •

edited

mikf commented Aug 14, 2022 •

edited

nisehime commented Aug 14, 2022

mikf commented Aug 14, 2022

biggestsonicfan commented Aug 14, 2022

nisehime commented Aug 14, 2022

biggestsonicfan commented Sep 3, 2022

nisehime commented Sep 3, 2022

biggestsonicfan commented Sep 3, 2022

nisehime commented Sep 3, 2022

biggestsonicfan commented Jan 24, 2023

wankio commented Jan 24, 2023

Large Twitter Galleries Not Fully Downloading #2226

Large Twitter Galleries Not Fully Downloading #2226

Comments

ghost commented Jan 25, 2022 • edited by ghost

Zoodee commented Jan 25, 2022

ImportTaste commented Jan 28, 2022

tux93 commented Feb 10, 2022

wankio commented May 25, 2022

Twi-Hard commented May 25, 2022

wankio commented May 25, 2022

biggestsonicfan commented Aug 9, 2022 • edited

nisehime commented Aug 11, 2022

biggestsonicfan commented Aug 14, 2022 • edited

mikf commented Aug 14, 2022

biggestsonicfan commented Aug 14, 2022

nisehime commented Aug 14, 2022 • edited

mikf commented Aug 14, 2022 • edited

nisehime commented Aug 14, 2022

mikf commented Aug 14, 2022

biggestsonicfan commented Aug 14, 2022

nisehime commented Aug 14, 2022

biggestsonicfan commented Sep 3, 2022

nisehime commented Sep 3, 2022

biggestsonicfan commented Sep 3, 2022

nisehime commented Sep 3, 2022

biggestsonicfan commented Jan 24, 2023

wankio commented Jan 24, 2023

ghost commented Jan 25, 2022 •

edited by ghost

biggestsonicfan commented Aug 9, 2022 •

edited

biggestsonicfan commented Aug 14, 2022 •

edited

nisehime commented Aug 14, 2022 •

edited

mikf commented Aug 14, 2022 •

edited