Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--since support for various commands for refresh-by-cron #20

Closed
simonw opened this issue Oct 13, 2019 · 3 comments
Labels

Comments

@simonw
Copy link
Collaborator

@simonw simonw commented Oct 13, 2019

I want to run a cron that updates my Twitter database every X minutes.

It should be able to retrieve the following without needing to paginate through everything:

  • Tweets I have tweeted
  • My home timeline (see #19)
  • Tweets I have favourited

It would be nice if this could be standardized across all commands as a --since option.

@simonw simonw added the enhancement label Oct 13, 2019
@simonw

This comment has been minimized.

Copy link
Collaborator Author

@simonw simonw commented Oct 13, 2019

For favourites a --stop_after=200 option is probably good enough.

simonw added a commit that referenced this issue Oct 15, 2019
simonw added a commit that referenced this issue Oct 16, 2019
Refs #20

Also added some initial rate limit error handling code.
simonw added a commit that referenced this issue Oct 16, 2019
@simonw

This comment has been minimized.

Copy link
Collaborator Author

@simonw simonw commented Oct 16, 2019

I'm not going to do this for "accounts that have followed me" and "new accounts that I have followed" - instead I will recommend running the friend_ids and followers_ids commands on a daily basis since that data doesn't really change much by the hour.

@simonw simonw closed this Oct 16, 2019
@simonw

This comment has been minimized.

Copy link
Collaborator Author

@simonw simonw commented Oct 21, 2019

In case anyone is interested, here's an extract from the crontab I'm running these under at the moment:

1,11,21,31,41,51 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite user-timeline /home/ubuntu/twitter.db -a /home/ubuntu/auth.json --since
2,7,12,17,22,27,32,37,42,47,52,57 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite home-timeline /home/ubuntu/timeline.db -a /home/ubuntu/auth.json --since
6,16,26,36,46,56 * * * * /home/ubuntu/datasette-venv/bin/twitter-to-sqlite favorites /home/ubuntu/twitter.db -a /home/ubuntu/auth.json --stop_after=50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.