Storage-MongoDB 2 steps loading

[j-coll]

• Define the stage collection schema
  • Document the stage collection schema in the wiki
• Implement loaders and readers from the stage collection
  • Avoid false DupKey exception with concurrent upserts. see MongoDB [SERVER-14322]
  • Read sorted. Split not overlapping batches.
• Implement new Variant loader reading from the stage collection
  • Detect overlapping variants
  • Fetch loaded variants for overlapping variants
  • Handle horizontally splitted files (e.g. divided per chromosome)
  • Add specific tests scanning MongoDB to check the documents.
  • Update VariantMongoDBWriterTest
• Add direct loader for the first file in a chromosome to load directly to the variants collection . see Add direct loader for the first file in a chromosome to load directly to the variants collection #354
• Implement batch load method using the new functionality : VariantStorageManager#index(List<URI>)
• Check resume mode
  • This will require to record the ongoing operations for staging and merging. Use BatchFileOperation
  • Resume mode for Stage loading step
  • Resume mode for Merge step
• Add batch load tests
• Concurrent merge from different studies. See "concurrentMerge"

[j-coll]

When updating the variants collection, there are tree possible scenarios:

• The document does not exists (newVariant)
• The document exists but the study doesn't (newStudy)
• The document and the study exists

The first approach will consist in making a mongodb.insert for each newVariant, and updates for the other operations.

Allowing concurrent merges from different studies, it may happen that a document, supposedly non existing in the collection (newVariant), to be inserted from several studies at the same time, causing a DupKey exception.

To sort this out, the easiest solution is to perform a mongodb.update with the param $setOnInsert.

This second approach will use only update operations, solving the first and the second scenarios at the same time.