Import optimizations

[jpmckinney]

(If the current algorithm proves to run too slowly)

If foreign objects are present:

• When dumping JSON objects, put the index in the filename, so that import order can match scraper order
  • e.g. if scraping an org. hierarchy with foreign objects, preserving order will ensure parents are imported first
• Create a dependency graph anyway and use it as a best-guess initial order
  • based on assumption that real-world input will be far from worst-case input
• Perform foreign object resolution against both the DB and the unimported objects
  • would need to use a unique temporary collection
• Add more information (what info?) to the models so that a reasoner can better plan the import
  • ... reasoning is hard ...

[jpmckinney]

So far, it's been possible to use foreign keys instead of foreign objects to avoid slow foreign object resolution.