Set up automated process to export pages/versions

[Mr0grog]

For people who want to do more complicated analyses offline, it might be helpful to dump just the "data" tables (i.e. not including permissions, users, etc)

[danielballan]

I see two use cases here:

• Exporting anonymized tables for manual off-line analysis and fiddling.
• Notifies a processing service of new pages/versions so that this service can assign a priority / filtering status to each Change. (This involves first computing diffs. Whether the processing service then caches those diffs for awhile or destroys them to save space is a separate question. Either seems plausible.)

[Mr0grog]

I was definitely focusing on the former.

I think the latter might be better handled by polling the API with a capture_time= query (or maybe this is a reason to add query support for created_at?). I guess we could alternatively have some sort of webhook for imports or have -db drop something onto a message queue of some sort that all the various services monitor (is that still a thing we’re planning?).

[danielballan]

Right... I'd say push is better than pull, but pull is easier to quickly hack together. I don't think we have to decide yet. The initial test can just grab everything and then we'll see if we want to tackle setting up a queue right away or not.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in seven days if no further activity occurs. If it should not be closed, please comment! Thank you for your contributions.