Skip to content
master
Switch branches/tags
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 

readme.md

Dockerized twitter scraper

This is a dockerized twitter scraper built on Python. It is composed of the following 5 micro-services:

  1. MongoDB (database)
  2. Mongo-express (database UI)
  3. RabbitMQ (message queue management)
  4. Nameko (microservices framework)
  5. Twitter scraper (actual scraper)

Environment variables

Using the docker-scraper-twitter.default.env file as reference, create a docker-scraper-twitter.env file in the same location with the following:

TWITTER_API_CONSUMER_KEY=***
TWITTER_API_CONSUMER_SECRET=***
TWITTER_API_ACCESS_TOKEN=***
TWITTER_API_ACCESS_TOKEN_SECRET=***
TWITTER_API_FILTER=keyword
AMQP_URI=amqp://guest:guest@rabbitmq

Config

Using the config.sample.py file as reference, create a config.py file in the same location, and customize its contents as needed.

Start the service

docker-compose up

RabbitMQ admin panel

http://localhost:15672

MongoDB admin panel

http://localhost:8081

About

Dockerized Twitter scraper written in Python, composed of 5 micro-services.

Topics

Resources

Releases

No releases published

Packages

No packages published