Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: Ability to specify a date to scrape back from #43

Closed
sipka opened this issue Dec 2, 2020 · 2 comments
Closed

Feature request: Ability to specify a date to scrape back from #43

sipka opened this issue Dec 2, 2020 · 2 comments

Comments

@sipka
Copy link

sipka commented Dec 2, 2020

Currently, when fetching Parlies based on hashtag / user, the results are returned from that moment in time, backwards. It would be very helpful, for research purposes, if a user would be able to specify the date to crawl back from.

For example, if one is investigating activity around elections time on #maga hashtag, right now it would take days of scraping to get to election time data, as this is a high activity hashtag.

I am not sure if this is easy to do or not, although the fact that system message seems to include some date is a positive sign (e.g. the startkey part of Fetching v1/post/hashtag?tag=maga&limit=10&startkey=2020-11-25T21%3A24%3A27.365Z_250881603 )

@castle-lemongrab
Copy link
Collaborator

Take a look at #42 (in progress) – I think this will achieve exactly what you're looking for. If not, let me know.

@sipka
Copy link
Author

sipka commented Dec 3, 2020

Exactly. Yes. Perfect. Thank you!

@sipka sipka closed this as completed Dec 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants