Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Bulk data can and should be created incrementally #293
it _just_ dawned on me that our new bulk data system can be incremental. The new process should be something like:
That last step is currently pretty hard, but it could be made pretty easy if we added a "Deleted" flag to the database. I'm tempted to do that anyway so
Anyway, however this is implemented, we could _drastically_ speed up bulk file generation, which, despite my efforts, is currently taking way too long.