NullReferenceException after upgrading to 2.2.6

[majastrz]

Description

After upgrading both MSTest.TestAdapter and MSTest.TestFramework to 2.2.6, some of my tests are failing with a NullReferenceException that appears internal to the test framework.

Steps to reproduce

1. Clone https://github.com/Azure/bicep
2. Checkout the majastrz/try-update-tests-again branch.
3. Run dotnet test from repo root.

Expected behavior

Tests pass like they do in the main branch.

Actual behavior

Many of the tests fail with the following exception (seen in the output of dotnet test):

  Failed GetPosition_ValidLineStartsAndOffset_ReturnsConvertedPosition (System.Collections.Immutable.ImmutableArray`1[System.Int32],0,(0, 0))
  Error Message:
   Object reference not set to an instance of an object.
  Stack Trace:
     at ReadArrayOfintFromJson(XmlReaderDelegator , XmlObjectSerializerReadContextComplexJson , XmlDictionaryString , XmlDictionaryString , CollectionDataContract )
   at System.Runtime.Serialization.Json.JsonCollectionDataContract.ReadJsonValueCore(XmlReaderDelegator jsonReader, XmlObjectSerializerReadContextComplexJson context)
   at System.Runtime.Serialization.Json.JsonDataContract.ReadJsonValue(XmlReaderDelegator jsonReader, XmlObjectSerializerReadContextComplexJson context)
   at System.Runtime.Serialization.Json.XmlObjectSerializerReadContextComplexJson.ReadDataContractValue(DataContract dataContract, XmlReaderDelegator reader)
   at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, Type declaredType, DataContract& dataContract)
   at System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserialize(XmlReaderDelegator xmlReader, Type declaredType, DataContract dataContract, String name, String ns)
   at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.InternalReadObject(XmlReaderDelegator xmlReader, Boolean verifyObjectName)
   at System.Runtime.Serialization.XmlObjectSerializer.InternalReadObject(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.XmlObjectSerializer.ReadObjectHandleExceptions(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.ReadObject(XmlDictionaryReader reader)
   at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.ReadObject(Stream stream)

Environment

Local repro was on a Win10 machine. Will update issue once my CI job finishes failing.

[Haplois]

Thank you for reporting this issue. It's caused by the changes on how we handle tests with data sources after version 2.2.3. Please update to version 2.2.6 and opt-out out of this behavior as described here.

[majastrz]

@Haplois Setting aside the breaking change, throwing a nullref is not a good experience. Can you detect this and output a good error message?

[Haplois]

These exceptions are generally because of serialization problems. We tried to detect all edge-cases we can think of, but we failed to assess the risk. Right now there is an opt-out from that feature. We'll be clarifying the error message as well.