Skip to content
Companion Repo for a Blog Post
Python Dockerfile Makefile
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.dockerignore
.gitignore
Dockerfile
Makefile
README.md
app.py
docker-compose.yml
requirements.txt

README.md

Batch Processing in Python using Dramatiq

This repo is a companion for a blog post on my website about a simple batch processing technique in Python using Dramatiq, Docker, and Redis. The technique is similar to a simple MapReduce-style of batch processing.

Run The Project

You will need the following installed:

  • GNU Make (As a task runner)
  • Docker
  • Docker Compose

Building the container is as simple as running make:

# That's all folks.
make

Then you can launch the stack (again with make):

make compose

Alternatively, you can just run docker commands directly.

About the Project

The script in app.py downloads Moby Dick, and tries to extract all person entities from the text using Spacy.

Because Dramatiq and Redis are used as a backend for processing the documents, it's possible to scale the processing out horizontally as far as required, to greatly reduce execution time.

License

MIT (c) 2019 Madelyn Eriksen

You can’t perform that action at this time.