You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Objective
We are currently pulling tweets on an ad-hoc basis. Since we need as much training data as possible to build good models, and because Twitter only gives us 7 days of data at any given time, we want to build a service that regularly pulls and saves data from Twitter.
First steps
In order of complexity, we will want to spin up a server on glitch.me, Heroku, or AWS EC2. The simplest implementation of this tweet puller would be a job that hits the Twitter API and dumps the response to a file. We can schedule this job using cron.
Useful tools
Crontab Man Pages - man7.org
Twitter API Docs
The text was updated successfully, but these errors were encountered:
@rileypredum Sure! I would love for you to take it on if you're interested, or I can also complete it if necessary. I'll probably work on the tasks that people don't think are as interesting.
@pahdo we also want to make sure the tweets are grabbing everything we want in its current form. Have you checked out my updates to the tween_scrape_and_clean notebook? It grabs more fields now.
Yep! That's a good point. I saw that work has been done on pulling tweets and metadata. I'll go ahead and set up the basic infrastructure to run some arbitrary job on a regular interval. We can probably just use your code when it's ready.
Key topics
Objective
We are currently pulling tweets on an ad-hoc basis. Since we need as much training data as possible to build good models, and because Twitter only gives us 7 days of data at any given time, we want to build a service that regularly pulls and saves data from Twitter.
First steps
In order of complexity, we will want to spin up a server on glitch.me, Heroku, or AWS EC2. The simplest implementation of this tweet puller would be a job that hits the Twitter API and dumps the response to a file. We can schedule this job using
cron
.Useful tools
Crontab Man Pages - man7.org
Twitter API Docs
The text was updated successfully, but these errors were encountered: