Determine how to trigger processing/analysis when new versions are added to DB

[Mr0grog]

Adding this to the main repo because it has potential implications for both -processing and -db.

Now that @janakrajchadha is starting to make some headway on analysis, we need to get serious about determining how and when to run that analysis against data that has been added to the DB.

Things that have been discussed:

1. Have processing tools poll DB on a regular interval looking for new data. (Highest overhead, but least effort; no major new systems or features have to be built for this.)

2. Create a message queue. DB should add items to the queue when page versions are imported. Processing tools will read from that queue. If we move processing to AWS, we can probably rely on Amazon SQS for this (if staying on GCloud, Cloud Pub/Sub). Otherwise, we’ll need to set up our own queueing service. ZeroMQ and Beanstalkd both seem like good options here because they are language-agnostic.

3. Webhooks. Processing tools could register themselves with DB as callable webhooks.

4. Other ideas?

[danielballan]

I think I like (2) best. It matches up to some early discussions on how we envisioned this might work, and it lets of deploy on one machine or many, as needed. I have some experience with ZeroMQ, so I'd be comfortable starting there, all else being equal.

[Mr0grog]

Agree (2) is the right way to be doing this. If we need to hook up something quickly, maybe worth starting with (1), but if we have the time, best to do it right with a queue from the start. 👍 on ZeroMQ.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in seven days if no further activity occurs. If it should not be closed, please comment! Thank you for your contributions.

[Mr0grog]

Heyyyyyyyy we did this!