Optimize Data Store for performance

[schnuerle]

Once the RDS is complete in Phase 1, we'd like to optimize it for performance. If you have ideas on how to do this with index or other methods, you can make a pull request to add support for it with examples to this repo so everyone can benefit. Use this issue as a place to discuss and collaborate.

[jrstill]

As discussed in #25, there are some pretty big performance gains to be had on insert-heavy workloads (such as when reprocessing lots of old files) if the FKs are removed. The larger the quantity of files in the queue, the greater the impact, as lock contention only gets worse the more there is. Just noting all of that here so we can review later.

[schnuerle]

I've got some scripts to remove all the FKs and some to add them all back in. I've been backloading data and what I do is remove the FKs and it's more than twice as fast, then add them in at the end.

Once all the data is backloaded (doing it in 1 month chunks now) I'd like to post scripts for creating indexes, which really speed up query analysis.

I'm working on writing these and documenting them and will make a PR to add the docs and scripts to the repo when done.

[schnuerle]

Working on this here: #41