How would I scrape "random" public posts between two dates?

[josh-ashkinaze]

For a project, I am interested in scraping random public posts between two dates. If I can't do random. then I am fine with searching via a proxy slike "the" or "of" or "a" etc.

Now, I know the app.bsky.feed.searchPosts endpoint can search posts by query---but is there a way to search for posts via two date ranges? Looks like that only takes in a query string as a parameter but not a date.

Any help is appreciated!

[marcomaroni-github]

I have post the same question here

[marcomaroni-github]

@josh-ashkinaze I think I've found a way.

Having the need to find all the posts of an account within a range of dates, I found a way by analyzing how the search parameters are translated into the API by the web app.
Here is an example:

curl --location 'https://public.api.bsky.app/xrpc/app.bsky.feed.searchPosts?q=from%3A%40mm-twitter-archive.bsky.social&since=2023-08-23T00%3A00%3A00.000Z&until=2023-08-23T23%3A59%3A59.000Z'

It appears to be an undocumented trick, the official documentation makes no mention of it:
https://docs.bsky.app/docs/api/app-bsky-feed-search-posts

[shaughnessyo]

Disclaimer: this might be a terrible way of doing this and I'm sure there are limitations I haven't found yet related to how postSearch works (for example, it seems like automated posts by bots are excluded from search), but it works for my low stakes purposes.

I wanted to download all of the posts for a given day in order to play with topic modeling over time-- I do data science and machine learning stuff mostly, but I haven't spent much time working with bulk text data. So I threw together a tool that breaks days into a set of postSearch since/until parameter time intervals, asynchronously fetches as many 1000 page blocks of posts as are in each interval by iteratively setting the until parameter value to the earliest time in the previous block of posts until it reaches the since value or the query returns no new posts, and then it does some cleanup and saves the results as json or inserts them into sqlite or postgres. I run it at most at half the rate limit because nothing I'm doing is time sensitive. I can clean it up and post it as a gist or something if anyone is interested.

[erlonL]

this really got me interested, can you share some snippet to do something like that? ty!

[shaughnessyo]

since I posted that, there have been changes to how searchposts pagination works (that is to say, it no longer seems to work), which makes my previous approach no longer viable.

I suspect it's probably still possible to download some-to-most of the posts from a specific day using created at/indexed at timestamps rather than pagination (break a day into say hour long intervals and query search posts for each hour since=hour:00:00 until=hour:59:59, which will return 100 posts, then you'd update the until= the timestamp of the earliest post in that batch, and run that iteratively until each hour doesn't return new posts). Of course, this might not actually work at all and I haven't tried it in part because now there's a new lighter weight json firehose: https://github.com/bluesky-social/jetstream and I feel like it's probably preferable that I work with the existing tools that the devs have made available.

It would still be useful in a mostly niche way to be able to download posts for a specific day. For topic modeling, it meant being able to get posts from specific days when I already knew the events that had happened to see how they were reflected on bluesky, which was useful for testing models because I sort of knew what to expect from them. But nothing I'm working on needs it and if I really did, I could just download all of the posts.