Discussion: Docker vs Pip vs Virtual Machine

[madisonb]

I would like to open up a discussion for Scrapy Cluster as to how it can be easier to work with.

As of right now, SC 1.1 (almost ready) allows you to do local development on a single Virtual Machine (at time of writing on the dev branch). This single VM is just to do local testing and should not be used in production.

This leaves you with a production deployment where a user must manually stand up Zookeeper, Kafka, and Redis at their desired scale and deploy SC to the various machines they want it to run on. This is done either manually by copying files, or (potentially) pip packages for the 3 main components. Ansible can help here to a degree but is quirky for your OS setup and is not always as modular.

Docker would allow you both the flexibility of deploying to arbitrary Docker Servers, and the ease of use of standing up your cluster. If we bundled the 3 main components as Docker Containers then with a bit of tweaking I think an easily scalable solution is possible. Especially using something like Mesos, Tumtum, Compose, or just plain Docker makes it really easy to run everything.

The downside to this is that it may be difficult for users to add custom spiders, pipelines, and middleware to their Scrapy based project, especially if heavy customization is going on and the spiders are deployed to a lot of different servers. ...Not to mention if the user adds custom plugins to the Kafka Monitor or Redis Monitor, would the user then would need to bundle their own docker container?

So the question is, what route seems like the most flexible, while allowing both local development and production scale deployment? What is the future of deploying distributed apps and how can we make SC extendable, flexible, deployable, and dev friendly?

[madisonb]

I think we should go down the Docker route, which brings us to a couple different choices:

1. A single docker image with Kafka, Redis, Zookeeper, and Scrapy Cluster. The container starts off with a certain commit/release of this repo in a specific folder. Container parameters would include the three folders that contain the Kafka Monitor, Redis Monitor, or the Scrapy project in order to allow someone to override or customize their setup. Other flags could also include number of spider processes to run within the container. This image is easy to work with because every single thing is self contained, but is probably going to be really big and bulky.
2. Rely on the user to set up Docker Kafka, Redis, and Zookeeper, and provide a single standalone Scrapy Cluster docker image containing the three core components. User still has the same container parameters to adjust # of spiders or folders, but would need to set up the ports correctly in order to work with the other containers.
3. Set up three different containers (each for Kafka Monitor, Redis Monitor, and Scrapers), which is basically splitting up item 2 above. This is useful for scaling and redundancy as the spiders are meant to scale really easily, so you just increase the number of spider containers.
4. Use either the concepts of Scrapy Cluster Docker images (either 1 single image or 3 different ones) and use docker compose to orchestrate them together. It is not compatible with Kitematic (see issue 137)) but would help new developers out by quickly allowing them to stand up a whole cluster of containers to test the functionality.

# 4 will allow us to potentially scale via Mesos or any other docker scaling solution, and is friendly enough to new users to allow them to configure their own code, test out a full scale deployment, and not have to fret about packaging or machine level configurations.

[madisonb]

Closing this since we are committing to Docker.