AV running System.Net.Http.Functional.Tests

[danmoseley]

https://ci.dot.net/job/dotnet_corefx/job/master/job/code_coverage_windows_prtest/17/consoleText

  
  Unhandled Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
     at Internal.Cryptography.Pal.ChainPal.MapRevocationFlags(X509RevocationMode revocationMode, X509RevocationFlag revocationFlag)
     at Internal.Cryptography.Pal.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, DateTime verificationTime, TimeSpan timeout) in D:\j\workspace\code_coverage---d5838fef\src\System.Security.Cryptography.X509Certificates\src\Internal\Cryptography\Pal.Windows\ChainPal.BuildChain.cs:line 71
     at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException) in D:\j\workspace\code_coverage---d5838fef\src\System.Security.Cryptography.X509Certificates\src\System\Security\Cryptography\X509Certificates\X509Chain.cs:line 118
     at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate) in D:\j\workspace\code_coverage---d5838fef\src\System.Security.Cryptography.X509Certificates\src\System\Security\Cryptography\X509Certificates\X509Chain.cs:line 105
     at System.Net.Http.WinHttpCertificateHelper.BuildChain(X509Certificate2 certificate, String hostName, Boolean checkCertificateRevocationList, X509Chain& chain, SslPolicyErrors& sslPolicyErrors) in D:\j\workspace\code_coverage---d5838fef\src\System.Net.Http.WinHttpHandler\src\System\Net\Http\WinHttpCertificateHelper.cs:line 31
     at System.Net.Http.WinHttpRequestCallback.OnRequestSendingRequest(WinHttpRequestState state) in D:\j\workspace\code_coverage---d5838fef\src\System.Net.Http.WinHttpHandler\src\System\Net\Http\WinHttpRequestCallback.cs:line 272
     at System.Net.Http.WinHttpRequestCallback.RequestCallback(IntPtr handle, WinHttpRequestState state, UInt32 internetStatus, IntPtr statusInformation, UInt32 statusInformationLength) in D:\j\workspace\code_coverage---d5838fef\src\System.Net.Http.WinHttpHandler\src\System\Net\Http\WinHttpRequestCallback.cs:line 94
     at System.Net.Http.WinHttpRequestCallback.WinHttpCallback(IntPtr handle, IntPtr context, UInt32 internetStatus, IntPtr statusInformation, UInt32 statusInformationLength) in D:\j\workspace\code_coverage---d5838fef\src\System.Net.Http.WinHttpHandler\src\System\Net\Http\WinHttpRequestCallback.cs:line 47

[danmoseley]

@steveharter @bartonjs can't tell from the log which test it was.

[steveharter]

Additional info: machine: win2012-97aa20 + a code coverage run.

It would be interesting to know:

1. Does the AV only happens on win2012? FWIW with win2012 I am able to repro random failures in System.Net.Http.Functional although those failures so far have resulted in various Http exceptions, not AVs.
2. Does the AV occur for non-code coverage runs? Could that be contributing?

Note that MapRevocationFlags() is a simple static method that takes two enums and returns another enum - no objects, native calls or unsafe code. Also there are no line numbers present for that method in the call stack, indicating a potential problem entering or exiting the method - I'll check to see how code coverage works w.r.t. gathering metrics per method.

[steveharter]

@stephentoub have you seen random AVs in code coverage runs? Who owns code coverage for corefx?

Could the code coverage be adding IL to methods that may cause AV under certain conditions? (perhaps with some other underlying memory or stack corruption)

[stephentoub]

have you seen random AVs in code coverage runs?

No, I haven't.

Who owns code coverage for corefx?

A variety of folks contribute to it, including me. Not sure anyone "owns" it.

Could the code coverage be adding IL to methods that may cause AV under certain conditions?

It uses the profiling APIs to add code that it uses to track whether each line and branch is hit. It's possible that's leading to issues, but we've not seen any problems with it, and we've been using this tool now for quite a while.

[danmoseley]

We're working on getting dumpfiles from test runs, so that may be the next step.

[karelz]

Dupe of dotnet/corefx#17127