Test "update" commands in deterministic correctness

[apkar]

Update tests are not currently being tested elaborately with deterministic correctness runs. We should enable them and this is to track issues found with that.

[dongxinEric]

Found the following issues that need to be resolved before we enable the update tests in correctness:

• Document not returned by the same query after update an array field that contains a None value. #135
  • We are moving this bug out of 1.7.0 given the fact that None values in arrays are rare. Will fix this soon after 1.7.0 milestone.
• $addToSet operator is not adding None values correctly #151

[dongxinEric]

So after spending almost a week trying to fix the update tests in correctness, I found it hard to got fully fixed up to achieve what it tries to currently achieve with following problems.

Problem 1: atomicity

• MongoDB's update is not atomic and, as far as I know, it applies updates until:
  • All updates are done, OR
  • one of the update triggers an error got thrown out
• However in DocLaye, since operations are grouped into 3 seconds' batches, the update works like:
  • All update are done, OR
  • one of the update triggers an error got thrown out and thus all updates in this batch are aborted

Thus in MM it's hard to do what DocLayer does since it has no knowledge of which update operations will be grouped together.

Problem 2: index

The index will decide which document got returned, and thus updated first in DocLayer. However, MM'd index does not actually affect the order of the document returned out of a query. Thus we would see different error returned in following case:

• There are more than one documents that would be updated and Doc1 and Doc2 will both throw error;
• In DocLayer, because of index, Doc1 got evaluated first and thus throw error 1;
• In MM, because the index dose not affect order, Doc2 got evaluated first and thus throw error 2.

Now we have a pair of mismatched errors. This maybe solved by disabling the exact error code match, but that undermines the value of the test itself, as in the correctness tests are supposed to be deterministic.

Possible workaround

Now one possible workaround would be:

• Only continue the current iteration of the test to check the consistency of the data in DocLayer and MM when there is no update error (like unique index violation and illegal update operator usage and so on) happened

This way we make sure in happy cases, the DocLayer does update the data correctly. However, this also means we might miss bugs in DocLayer which only happens when there is update errors happens.

I will open a separate issue tracking the re-enable the tests in failure cases, but outside of the 1.7.0 scope.

[dongxinEric]

Update tests are now enabled in nightly.