Support for --until

[ppival]

Another tool I have used, GetOldTweets3, supports not only the --since DATE flag, but also --until DATE, allowing a twitter scrape to focus between two dates. Syntax simply looks like --since 2015-05-24 --until 2015-08-23

Would it be possible to enable this for snscrape?

[JustAnotherArchivist]

It could be implemented, but it wouldn't really gain anything. --since is scraper-independent; the way it works is that it exits when reaching a result before the specified date, preventing paginating through potentially many older results and shortening the scrape. If I were to add --until, it would merely suppress the output of items newer than some date, but in the background, it would still have to fetch all results newer than that as well. So you don't gain any time or resources.

However, for Twitter specifically, you can use Twitter's proper since and until search filters. For example, snscrape twitter-search 'foobar since:2019-01-01 until:2020-01-01' would return tweets containing 'foobar' from the year 2019. since:YYYY-MM-DD produces tweets that were posted on that date or later; until:YYYY-MM-DD produces tweets that were posted before (but not on!) that date. (This used to be documented on the Twitter website, but they removed that since. You can find some more here by clicking on 'operators' below the search field. That list was also incomplete though.)

[Tachevaz]

Thank you for this great tool, it's a savior!

About the "since" argument - is there a way to implement it in a script as well, as an argument to snscrape.modules.twitter.TwitterUserScraper()? Or should it be added as a condition after snscrape.modules.twitter.TwitterUserScraper() has finished running?

[JustAnotherArchivist]

There are two ways to do that (without wasting time scraping things you're not actually interested in):

1. You can check the date of the Tweet objects returned by the scraper and break the loop when it's older than the time you're interested in. This is exactly what --since in the CLI does.
2. The user scraper is actually just a fancy wrapper around the much more general search scraper. So you could use that instead and rely on Twitter's date filters: snscrape.modules.twitter.TwitterSearchScraper('from:username since:2020-01-01')

[Tachevaz]

It worked, thank you so much!:))

[JustAnotherArchivist]

Closing this; see my first comment for the rationale.

[ppival]

Any reason this works for twitter-search, but not for twitter-user? The following just appears to sit there forever:
snscrape twitter-user "barackobama since:2015-09-10 until:2015-09-12” > baracktweets.txt

TIA! You, @JustAnotherArchivist, are just about the most patient and responsive maintainer I've ever come across! :-)

It could be implemented, but it wouldn't really gain anything. --since is scraper-independent; the way it works is that it exits when reaching a result before the specified date, preventing paginating through potentially many older results and shortening the scrape. If I were to add --until, it would merely suppress the output of items newer than some date, but in the background, it would still have to fetch all results newer than that as well. So you don't gain any time or resources.

However, for Twitter specifically, you can use Twitter's proper since and until search filters. For example, snscrape twitter-search 'foobar since:2019-01-01 until:2020-01-01' would return tweets containing 'foobar' from the year 2019. since:YYYY-MM-DD produces tweets that were posted on that date or later; until:YYYY-MM-DD produces tweets that were posted before (but not on!) that date. (This used to be documented on the Twitter website, but they removed that since. You can find some more here by clicking on 'operators' below the search field. That list was also incomplete though.)

[JustAnotherArchivist]

twitter-user is actually just a wrapper around twitter-search using the search term from:username (plus code to extract user information from the profile page). So: snscrape twitter-search 'from:barackobama since:2015-09-10 until:2015-09-12'

Happy to help! I'm glad this work is finally getting some use outside of the immediate use case I wrote it for. :-)

[Uzair1947]

can we give time to until and since

[JustAnotherArchivist]

@Uzair1947 #259