Unable to scrape all the data of an Twitter account, need help in scraping all of its contents #4618
-
Hello everyone, a recent problem occurred to me while I was using gallery-dl on Twitter. Regarding what config I am using for Twitter, I am using the config from #4534, with afaik no altercations to it yet. I hope someone is willing to assist me on this matter. thanks 👍 the relevant config.json parts
|
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 6 replies
-
Friend, you may want to double-check those credentials before you publish your username and password on a public forum. Hope you've got 2FA turned on. |
Beta Was this translation helpful? Give feedback.
-
I'm not sure what's the etiquette for questions that aren't answered in a couple weeks but I'll make an input requesting for help again. I recently got my g-dl up to date to the latest version (I figured out that I haven't updated it in a while) and retried scraping from the same account as above again, however, it still has the same problem, although this time it stopped downloading when it reached 3215 items. I hope that someone with better insights into g-dl might be able to assist me in this matter. Thank you. |
Beta Was this translation helpful? Give feedback.
-
If you're hitting a limit with gallery-dl on Twitter after scraping around 3k images, despite there being more, it sounds like you might be encountering API rate limits or the tool's internal limitations. The config from #4534 should generally work well, but every tool has its constraints, especially with large volumes of data. For overcoming such limitations, Crawlbase could offer a potential solution. While it's not specifically designed for Twitter scraping, it's robust handling of requests and data could help manage and possibly bypass the stumbling blocks you're facing. It's particularly useful for large-scale scraping projects where managing requests efficiently and avoiding detection become crucial. You can also try brighdata or scrapingbee as alternatives. |
Beta Was this translation helpful? Give feedback.
TLDR: Twitter search bad, but the only option
As far as I know, it is not really possible to get more items from a Twitter user than gallery-dl already gets, unless maybe with a more precise search query.
The current strategy is using the
/media
timeline for the first ~1000 items, followed by a search forfrom:USERNAME maxid:LAST_MEDIA_ID
. The problem with this is 1) there are no other ways to get older Tweets other than searching and 2) search got severely crippled, rate limited, and is inaccurate (i.e. it misses Tweets).Your config settings shouldn't matter all that much or at all as long as you run gallery-dl with
twitter.com/USERAME
as input URL.