Support for scraping tweet based on its ID

[santoshbs]

I was wondering if we could use snscrape to fetch the tweet text for a given twitter status id or a set of status ids. I could not find this in the documentation.

[JustAnotherArchivist]

There is no explicit support for this at the moment, but I intend to add something that also covers #51 and a reimplementation of #12 (cf. #98 (comment)).

As a workaround, it's possible to fetch individual tweets through the search using the since_id and max_id operators. However, you can't use those on their own. But this works, for example:

snscrape --jsonl twitter-search 'since_id:1320246372008853503 max_id:1320246372008853504 filter:safe'

If that produces no result, put a - before the filter:

snscrape --jsonl twitter-search 'since_id:1303506596216045567 max_id:1303506596216045568 -filter:safe'

The max_id is the ID of the tweet of interest, and the since_id is one below that; or in other words, since_id filters for tweets newer than an ID (not inclusive) and max_id filters for tweets older than an ID (inclusive).

Note that this is quite inefficient since you have to run a separate search for every tweet ID. If you have a lot of tweets, this is probably not a good idea. However, my upcoming implementation likely won't fare any better.

[maclean-alexander]

Hello, I've been looking for methods to get tweets from a list of IDs and this seems like a workable solution. However, I've noticed that from the list I've been using, less than 20% of the queries return tweet information. Doing manual checking for the IDs, most of them do exist, and I think the differentiating factor is that the tweets not being returned seem to be Retweets. I couldn't find anything that was filtering out retweets, so I was wondering if you knew of a way to get access to the tweets even when they are retweets. I had also tried looking at TwitterThreadScraper, but could not get that working in any capacity. Thank you!

[JustAnotherArchivist]

@maclean-alexander See #83 and #8. Basically, it isn't possible to reliably extract information on retweets because Twitter's search doesn't return them. I have not yet found time to look into implementing this, so no idea whether/how this will work with regards to retweets. Also, yes, TwitterThreadScraper is broken (#98 (comment)).

[maclean-alexander]

Ah ok, thank you for the quick response.

[clohaa]

How can I scrap data using tweet id with the updated module?

[JustAnotherArchivist]

CLI: snscrape twitter-tweet 20
Python: snscrape.modules.twitter.TwitterTweetScraper(20)

[Swati-Maste]

Hey, I want to know, how to get the tweet using the tweet ID

[Swati-Maste]

I tried the above python code, it says : AttributeError: module 'snscrape.modules.twitter' has no attribute 'TwitteTweetScraper'

[TheTechRobo]

Please make sure you're running Python 3.8 or later and the latest version of snscrape.

[Swati-Maste]

Thanks for the quick response. Yeah I did. But still doesn't seem to work.

[TheTechRobo]

You spelled it wrong, it's TwitterTweetScraper, not TwitteTweetScraper

[Swati-Maste]

oh yeah, thanks a lot

[Swati-Maste]

AttributeError: module 'snscrape.modules.twitter' has no attribute 'TwitterTweetScraper', I'm still getting this error