A bot/worker/cron/process to fetch & curate all the relevant news content from the web based on the user's subscription.
This service is responsible for fetching the news related to the keywords that the users have subscribed to. The list of all keywords that users have subscribed to are present in the database.
- The keywords are first fetched
- Multiple
workersare created to to fetch news concurrently - The keyqords are queued for the workers to pickup
- Once a worker picks up a keyword, it will fetch the news from newsapi.org's REST api
- Once the news is fetched, they are pushed to the database and the titles are cached locally. Because, the next time when the news is fetched, the same news should not be pushed to the dB again and again
This procedure is run multiple times in a day. Keep in mind, only the "new" news is sent to the database.
- First, fork and clone the repo. This will create a new folder named
news-api. - Navigate to
/news-apiand run:
go get -d
go build
./news-api
Read this
